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provides  a  two-page  analysis  of  the  largest  defense  programs.  This  research  project  examines  eight  major 
DOD  weapon  systems  using  the  yearly  published  GAO  reports  from  200372011.  The  purpose  of  this 
research  is  to  understand  what  risk  management  was  being  performed  on  those  programs,  how  it  was 
implemented,  when  it  was  implemented,  and  how  effective  it  was.  The  desired  outcome  is  to  make 
recommendations  on  improvements  to  the  risk  management  process  recommended  by  the  Risk 
Management  Guide  for  DoD  Acquisition  (DoD  2006)  More  specifically,  this  research  paper  addresses  risk 
mitigation  activities  that  are  considered  best  practices  and  that  are  documented  in  the  yearly  GAO 
Assessments  of  Selected  Weapon  Programs  reports.  Changes  in  cost,  schedule  and  performance  have  been 
tracked  over  time,  which  provides  a  unique  perspective  to  understanding  program  execution.  It  is  not 
possible  to  examine  every  aspect  of  eight  programs  over  a  nine-year  period.  This  project  has  focused  on 
some  key  systems  engineering  principles  that  are  risk-mitigation  activities.  The  activities  studied  are 
acquisition  strategy,  including  acquisition  phases,  milestones,  technical  reviews,  and  major  decision 
reviews.  Also  evaluated  were  technical  maturity,  design  maturity,  earned  value,  production  maturity,  and 
software  design.  An  effective  program  acquisition  strategy  reduces  program  risks.  Breaking  the 
development  into  phases  lowers  the  chances  of  designing  or  fielding  a  system  that  is  immature.  The 
milestone  reviews  are  used  to  evaluate  the  system  and  to  hold  back  programs  that  aren?t  mature  enough  to 
proceed  to  the  next  phase.  Technical  reviews  are  used  to  measure  the  progress  of  the  program  development 
and  are  event  based  to  ensure  the  timely  completion  of  the  development  effort.  Developmental  and 
operational  tests  are  used  to  verify  system  performance  before  large  sums  of  money  are  spent  on 
production  units.  Technology  Readiness  Levels  (TRLs)  are  a  measure  used  by  the  Department  of  Defense 
to  assess  the  maturity  of  an  evolving  technology  prior  to  incorporating 
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Abstract 


DoD  major  weapon  systems  historically  have  been  over  budget,  behind 
schedule,  and  typically  under  performing  in  terms  of  suitability.  The  Secretary  of 
Defense  proposed  a  fiscal  year  (FY)  2010  budget  that  ended  or  curtailed  all  or  part 
of  at  least  a  half  dozen  major  defense  acquisition  programs  that  were  over  cost, 
behind  schedule,  or  no  longer  suited  to  meet  the  warfighters’  current  needs. 

The  Government  Accountability  Office  (GAO)  publishes  a  major  DoD  weapon 
systems  report  every  March.  The  GAO  spends  a  considerable  amount  of  time 
evaluating  several  dozen  major  weapon  system  programs  and  provides  a  two-page 
analysis  of  the  largest  defense  programs. 

This  research  project  examines  eight  major  DOD  weapon  systems  using  the 
yearly  published  GAO  reports  from  2003-201 1 .  The  purpose  of  this  research  is  to 
understand  what  risk  management  was  being  performed  on  those  programs,  how  it 
was  implemented,  when  it  was  implemented,  and  how  effective  it  was.  The  desired 
outcome  is  to  make  recommendations  on  improvements  to  the  risk  management 
process  recommended  by  the  Risk  Management  Guide  for  DoD  Acquisition  (DoD, 
2006) 

More  specifically,  this  research  paper  addresses  risk  mitigation  activities  that 
are  considered  best  practices  and  that  are  documented  in  the  yearly  GAO 
Assessments  of  Selected  Weapon  Programs  reports.  Changes  in  cost,  schedule, 
and  performance  have  been  tracked  over  time,  which  provides  a  unique  perspective 
to  understanding  program  execution. 

It  is  not  possible  to  examine  every  aspect  of  eight  programs  over  a  nine-year 
period.  This  project  has  focused  on  some  key  systems  engineering  principles  that 
are  risk-mitigation  activities.  The  activities  studied  are  acquisition  strategy,  including 
acquisition  phases,  milestones,  technical  reviews,  and  major  decision  reviews.  Also 
evaluated  were  technical  maturity,  design  maturity,  earned  value,  production 
maturity,  and  software  design. 
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An  effective  program  acquisition  strategy  reduces  program  risks.  Breaking  the 
development  into  phases  lowers  the  chances  of  designing  or  fielding  a  system  that  is 
immature.  The  milestone  reviews  are  used  to  evaluate  the  system  and  to  hold  back 
programs  that  aren’t  mature  enough  to  proceed  to  the  next  phase.  Technical  reviews 
are  used  to  measure  the  progress  of  the  program  development  and  are  event  based 
to  ensure  the  timely  completion  of  the  development  effort.  Developmental  and 
operational  tests  are  used  to  verify  system  performance  before  large  sums  of  money 
are  spent  on  production  units. 

Technology  Readiness  Levels  (TRLs)  are  a  measure  used  by  the  Department 
of  Defense  to  assess  the  maturity  of  an  evolving  technology  prior  to  incorporating 
that  technology  into  a  system.  Unfortunately,  this  research  found  that  they  are  not  a 
good  indicator  of  program  performance.  Six  of  the  eight  programs  studied  identified 
their  technologies  as  mature  even  though  the  programs  experienced  significant 
delays  and  cost  overruns  after  the  technologies  had  been  identified  as  mature. 

In  the  GAO  reports,  design  maturity  was  based  on  the  number  of  released 
drawings.  Some  of  the  values  are  questionable  because  they  are  exact  multiples  of 
10%.  In  many  cases,  the  percent  of  drawings  decreased  after  obtaining  98%  or 
higher  in  a  previous  year.  This  suggests  that  much  redesign  had  taken  place,  that 
the  measurement  of  the  original  value  was  poor,  and/or  that  the  original  drawing  was 
released  before  the  design  had  stabilized. 

Earned  value  management  (EVM)  is  a  project  management  technique  for 
measuring  project  performance  and  progress  in  an  objective  manner.  Earned  value 
was  required  on  all  of  the  programs  in  the  GAO  reports  because  of  their  size. 
However,  only  a  few  programs  mentioned  EVM  in  the  GAO  reports  (only  22  times  in 
nine  years).  Even  when  mentioned,  the  data  were  often  vague.  In  some  cases,  the 
EVM  system  was  identified  as  broken.  Therefore,  EVM  was  not  an  effective  risk 
management  tool. 

Using  Statistical  Process  Control  (SPC)  tools  is  a  best  practice  for  managing 
quality  in  a  manufacturing  environment.  It  is  a  data-driven  method  for  monitoring  and 
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controlling  the  manufacturing  processes  that  affect  product  quality.  The  GAO  reports 
focused  on  the  use  of  SPC,  which  was  seldom  used  by  contractors.  Therefore,  in 
most  cases  the  risks  of  manufacturing  problems  could  not  be  addressed.  Production 
readiness  reviews  are  also  a  best  practice  and  should  be  used  to  address 
production  risks.  The  production  readiness  review  assesses  the  maturity  of  the 
design  for  going  into  production  and  can  be  an  early  indicator  of  future 
manufacturing  problems. 

The  GAO  reports  often  neglected  the  role  of  software  development  in  an 
acquisition  program.  Design  maturity  is  not  just  the  number  of  drawings  released. 
That  is  only  a  measure  of  the  physical  hardware  design.  Software  is  usually  very 
important  and  risky  on  large  weapon  programs.  Any  company  that  claims  to  be 
CMMI  Level  2  or  higher  will  be  using  metrics  to  manage  their  software  projects.  So, 
theoretically,  performance  metrics  should  exist  and  should  be  reported  to  manage 
the  risk  of  software  development  problems. 

Keywords:  DoD  major  weapon  systems,  Technology  Readiness  Levels 
(TRLs),  Earned  Value  Management  (EVM),  Statistical  Process  Control  (SPC) 
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ACQ  451 . “Integrated  Acquisition  for  Decision  Makers”  course  taught  by  DAL) 

ACQ  452 . “Forging  Stakeholder  Relationships”  course  taught  by  DAL) 

ANSI/EIA . American  National  Standards  Institute/Electronic  Industries  Alliance 

AT&L . Acquisition,  Technology,  and  Logistics 

CDR . Critical  Design  Review 

COTS . Commercial  Off-The-Shelf 

C-5 . An  Air  Force  transport  aircraft 


C-5  RERP  ....The  Air  Force’s  C-5  RERP  is  one  of  two  major  upgrades  for  the  C-5. 


The  RERP  is  designed  to  enhance  the  reliability,  maintainability,  and 
availability  of  the  C-5  by  replacing  the  propulsion  system;  modifying  the 
mechanical,  hydraulic,  avionics,  fuel,  and  landing  gear  systems;  and 
making  required  structural  modifications. 

CMMI . Capability  Maturity  Model  Integration 

DAG . Defense  Acquisition  Guidebook 

DAL) . Defense  Acquisition  University 

DCMA . Defense  Contract  Management  Agency 

DoD . Department  of  Defense 

DoDD . Department  of  Defense  Directive 

DT&E . Developmental  Test  and  Evaluation 

ECP . Engineering  Change  Proposal 

EDM . Engineering  Demonstration  Model 

EFV . Expeditionary  Fighting  Vehicle  (EFV) — The  Marine  Corps’  EFV  is 

designed  to  transport  troops  from  ships  offshore  to  inland  locales  at 
higher  speeds  and  from  longer  distances  than  its  predecessor,  the 


Assault  Amphibious  Vehicle  7A1  (AAV  7A1 ). 


EVM . Earned  Value  Management 

Excalibur . Excalibur  is  a  family  of  global  positioning  system-based,  fire-and- 

forget,  155-mm  cannon  artillery  precision  munitions  intended  to  provide 
improved  range  and  accuracy  being  developed  by  the  Army. 

FRP . Full-Rate  Production 

FY . Fiscal  Year 

GAO  . General  Accounting  Office  (before  July  7,  2004) 

GAO . Government  Accountability  Office  (after  July  7,  2004) 

GPQ . Group  Process  Questionnaire 
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Global  Hawk. The  Air  Force’s  Global  Hawk  is  a  high-altitude,  long-endurance 


unmanned  aircraft  with  integrated  sensors  and  ground  stations 
providing  intelligence,  surveillance,  and  reconnaissance  capabilities. 

Ho . Null  Hypothesis 

Hi . Alternate  Hypothesis 

IOC . Initial  Operational  Capability 

IPPD  . Integrated  Product  and  Process  Development 

IPT  . Integrated  Product  Team 

JSF . Joint  Strike  Fighter  (JSF) — The  DOD’s  JSF  program  is  developing  a 

family  of  stealthy,  strike  fighter  aircraft  for  the  Navy,  Air  Force,  Marine 
Corps,  and  U.S.  allies,  with  the  goal  of  maximizing  commonality  to 
minimize  life-cycle  costs. 

JTRS  . Joint  Tactical  Radio  System  (JTRS;  Cluster  1) — This  DOD  program  is 

developing  software-defined  radios  that  will  interoperate  with  selected 
radios  and  increase  communications  and  networking  capabilities. 

LRIP . Low-Rate  Initial  Production 

MRL . Manufacturing  Readiness  Level 

OT&E . Operational  Test  and  Evaluation 

PMT  401 . Program  Manager’s  course  taught  by  DAU 

PRR . Production  Readiness  Review 

SPC . Statistical  Process  Control 

SBIRS . Space  Based  Infrared  System  (SBIRS)  High — The  Air  Force’s  SBIRS 

High  satellite  system  is  being  developed  to  replace  the  Defense 
Support  Program  and  perform  a  range  of  missile  warning,  missile 
defense,  technical  intelligence,  and  battle-space  awareness  missions. 
Reaper . The  Air  Force’s  MO-9  Reaper  is  a  multirole,  medium-to-high-altitude 


endurance  unmanned  aerial  vehicle  system  capable  of  flying  at  higher 
speeds  and  higher  altitudes  than  its  predecessor,  the  MO-1  Predator 
A. 


R&D . Research  and  Development 

RDT&E . Research,  Development,  Test  and  Evaluation 

TRL . Technology  Readiness  Level 


USD(AT&L)..  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
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I. 


Introduction 


The  hypothesis  of  this  research  project  is  that  when  risks  increase  during 
development,  they  need  to  be  continuously  reflected  in  cost,  schedule,  and 
performance.  This  is  not  a  radically  new  idea.  The  DoD  and  contractors  treat  risk  as 
a  component  of  cost,  schedule,  and  performance  when  the  government  issues  a 
request  for  proposal  (RFP)  and  the  contractor  submits  a  bid. 

A.  Problem  Statement 

DoD  major  weapon  systems  historically  have  been  over  budget,  behind 
schedule,  and  typically  under  performing  in  terms  of  suitability.  The  2010  GAO  report 
on  major  weapon  systems  (GAO-1 0-388SP)  stated, 

The  Secretary  of  Defense  proposed  a  fiscal  year  2010  budget  that  ended  or 
curtailed  all  or  part  of  at  least  a  half  dozen  major  defense  acquisition 
programs — such  as  the  Air  Force’s  F-22A  Raptor,  the  Army’s  Future  Combat 
System,  the  Navy’s  DDG  1000  destroyer,  and  the  Missile  Defense  Agency’s 
Multiple  Kill  Vehicle — that  were  over  cost,  behind  schedule,  or  no  longer 
suited  to  meet  the  warfighters’  current  needs,  (p.  1) 

On  the  subject  of  technical  maturity,  the  report  (GAO,  2010)  stated, 

While  the  design  knowledge  of  DOD  programs  at  the  system-level  critical 
design  review  has  increased  since  2003,  these  programs  are  still  not  regularly 
demonstrating  that  these  designs  can  meet  performance  requirements  by 
testing  integrated  prototypes  before  the  critical  design  review — a  best 
practice.  ...  Of  the  33  programs  that  reported  that  they  either  had  tested  or 
were  going  to  test  an  early  system  prototype  and  provided  a  critical  design 
review  date,  only  4  did  so  before  their  critical  design  review.  The  remaining 
programs  tested  or  will  test  their  prototype,  on  average,  31  months  after  their 
critical  design  review.  While  few  programs  test  integrated  prototypes  by  the 
critical  design  review,  DOD  programs  are  testing  prototypes  earlier.  ...  (p.  15) 

On  the  subject  of  software  growth, 

Many  programs  are  at  risk  for  cost  growth  and  schedule  delays  because  of 
software  development  issues.  We  reported  in  our  last  assessment  that 
programs  experiencing  more  than  a  25  percent  growth  in  software  lines  of 
code  since  development  start  had  higher  development  cost  growth  and  longer 
schedule  delays  than  other  programs.  Seventeen  of  the  28  programs  that 
reported  data  on  software  lines  of  code  estimated  that  the  number  of  lines  of 
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code  required  for  the  system  to  function  has  grown  or  will  grow  by  25  percent 
or  more — up  from  14  programs  in  our  last  assessment.  Overall,  the  average 
lines  of  code  growth  or  planned  growth  for  the  28  programs  was  about  92 
percent,  (p.  20) 

B.  Purpose  of  this  Study 

This  research  project  examines  eight  major  DOD  weapon  systems  using 
yearly  published  GAO  reports.  The  purpose  of  this  research  is  to  understand  what 
risk  management  was  being  performed  on  those  programs,  how  it  was  implemented, 
when  it  was  implemented,  and  how  effective  it  was.  The  desired  outcome  is  to  make 
recommendations  on  improvements  to  the  risk  management  process  recommended 
by  the  Risk  Management  Guide  for  DoD  Acquisition  (DoD,  2006).  This  effort  is 
Phase  I  of  a  multi-phase  research  project  (Naval  Postgraduate  School,  BAA  Number 
NPS-BAA-1 1-02). 

C.  Overview  of  the  Research  Methodology 

This  research  project  is  based  on  the  GAO’s  2003-201 1  reports  on  major 
weapon  systems.  Eight  different  programs  were  identified  for  study.  The  yearly 
accomplishments  have  been  used  in  this  research  project  to  build  a  historical  record 
of  each  program.  The  historical  records  have  been  used  to  assess  the  effectiveness 
of  risk  management  activities  from  prior  years.  The  data  answer  the  questions, 

“What  was  planned  and  what  actually  happened?” 

D.  Research  Questions 

Research  questions  include  the  following: 


■  Can  the  effectiveness  of  risk  management  be  improved  by 
continuously  incorporating  risk  in  a  program’s  cost,  schedule,  and 
performance  objectives  and  plans? 

■  How  is  systems  engineering  being  used  to  manage  risk? 

■  How  effective  is  risk  mitigation  in  managing  large  weapon  system 
developments? 
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E.  Research  Hypothesis 


The  hypothesis  of  this  research  project  is  that  when  risks  increase  during 
development,  they  need  to  be  continuously  reflected  in  cost,  schedule,  and 
performance. 

F.  Objectives  and  Outcomes 

The  desired  outcome  is  to  make  recommendations  on  improvements  to  the 
risk  management  process  recommended  by  the  Risk  Management  Guide  to  DoD 
Acquisition  (DoD,  2006). 

G.  Limitations  of  the  Study 

The  research  project  is  based  on  GAO  reports  for  major  DoD  weapon  system 
acquisition  programs.  It  may  not  apply  to  smaller  DoD  programs. 

H.  Validity  of  the  Research 

The  research  is  based  on  GAO  reports  that  are  based  on  an  analysis  of  major 
DoD  acquisition  programs.  The  research  is  directly  applicable  for  DoD  programs. 
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Literature  Review 


This  research  project  is  based  on  written  reports  from  the  Government 
Accountability  Office  (GAO)  on  major  DoD  weapon  systems.  The  GAO  reports  come 
out  in  March  of  every  year  and  generally  provide  a  two-page  analysis  of  the  largest 
major  defense  programs.  The  GAO  reports  exist  from  2003-201 1 ,  although  the  first 
report  (GAO,  2003)  covered  a  smaller  number  of  programs  than  the  reports 
published  from  2004-201 1 .  The  following  are  the  GAO  reports  reviewed  in  this 
research  project: 

■  Defense  Acquisitions:  Assessments  of  Major  Weapon  Programs,  GAO- 
03-476  (May  2003) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-04-248  (March  2004) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-05-301  (March  2005) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-06-391  (March  2006) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-07-406SP  (March  2007) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-08-467SP  (March  2008) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-09-326SP  (March  2009) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-1 0-388SP  (March  2010) 

■  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-1 1 -233SP  (March  2011) 

The  purpose  of  the  GAO  reports  was  stated  in  the  201 1  report,  GAO-1 1- 
233SP: 

The  report  is  in  response  to  the  mandate  in  the  joint  explanatory  statement  to 
the  DOD  Appropriations  Act,  2009.  It  includes  observations  on  the 
performance  of  DOD’s  2010  portfolio  of  98  major  defense  acquisition 
programs;  data  on  selected  factors  that  can  affect  program  outcomes;  an 
assessment  of  the  knowledge  attained  by  key  junctures  in  the  acquisition 
process  for  a  subset  of  40  programs,  which  were  selected  because  they  were 
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in  development  or  early  production;  and  observations  on  the  implementation 
of  acquisition  reforms.  To  conduct  this  review,  GAO  analyzed  cost,  schedule, 
and  quantity  data  from  DOD’s  Selected  Acquisition  Reports  and  collected 
data  from  program  offices  on  performance  requirements  and  software 
development;  technology,  design,  and  manufacturing  knowledge;  and  the 
implementation  of  DOD’s  acquisition  policy  and  acquisition  reforms.  GAO  also 
compiled  one-  or  two-page  assessments  of  71  weapon  programs,  (p.  0) 

Analysis  of  the  GAO  reports  consisted  of  examining  the  write-up  for  each 

program  over  the  nine-year  period,  looking  for  trends  in  the  data,  identifying  risk- 

mitigation  activities,  and  evaluating  the  overall  effectiveness  of  the  risk  management 

process. 

Programs  that  were  studied  in  this  research  project  had  write-ups  for  most  of 
the  years  studied  (2003-201 1 ).  In  addition,  programs  were  selected  to  cover  the 
following  characteristics: 


■  the  largest  program  ever  in  the  DoD; 

■  a  program  with  a  very  small  number  of  production  units  (<  6); 

■  a  large  program  with  COTS  as  a  critical  component; 

■  a  large  joint  program; 

■  a  program  that  has  little  software;  and 

■  several  programs  in  which  software  is  critical  to  the  success  of  the 
program. 


Eight  programs  were  selected  for  review.  One  is  an  Army  system,  one  is  a 
Marine  Corps  system,  four  are  Air  Force  Systems,  and  two  are  joint  systems.  The 
systems  selected  went  through  the  acquisition  life  cycle  from  technology 
development  until  low-rate  initial  production  and  operational  test  and  evaluation. 
Some  reached  full  rate  production.  The  programs  that  were  selected  for  study  are 
the  following: 


1 .  Excalibur  (Army):  Excalibur  is  a  family  of  global  positioning  system- 
based,  fire-and-forget,  155-mm  cannon  artillery  precision  munitions 
intended  to  provide  improved  range  and  accuracy. 


NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 


-6  - 


2.  Expeditionary  Fighting  Vehicle  (EFV)1:  The  Marine  Corps’  EFV  is 
designed  to  transport  troops  from  ships  offshore  to  inland  locales  at 
higher  speeds  and  from  longer  distances  than  its  predecessor,  the 
Assault  Amphibious  Vehicle  7A1  (AAV  7A1 ). 

3.  C-5  Reliability  Enhancement  and  Reengining  Program  (RERP):  The 
Air  Force’s  C-5  RERP  is  one  of  two  major  upgrades  for  the  C-5.  The 
RERP  is  designed  to  enhance  the  reliability,  maintainability,  and 
availability  of  the  C-5  by  replacing  the  propulsion  system;  modifying  the 
mechanical,  hydraulic,  avionics,  fuel,  and  landing  gear  systems;  and 
making  required  structural  modifications. 

4.  Joint  Tactical  Radio  System  (JTRS;  Cluster  1  )2:  This  DoD  program  is 
developing  software-defined  radios  that  will  interoperate  with  selected 
radios  and  increase  communications  and  networking  capabilities. 

5.  Joint  Strike  Fighter  (JSF):  The  DoD’s  JSF  program3  is  developing  a 
family  of  stealthy,  strike  fighter  aircraft  for  the  Navy,  Air  Force,  Marine 
Corps,  and  U.S.  allies,  with  the  goal  of  maximizing  commonality  to 
minimize  life-cycle  costs. 

6.  Reaper:  The  Air  Force’s  MQ-9  Reaper  is  a  multirole,  medium-to-high- 
altitude  endurance  unmanned  aerial  vehicle  system  capable  of  flying  at 
higher  speeds  and  higher  altitudes  than  its  predecessor,  the  MQ-1 
Predator  A. 

7.  Global  Hawk4:  The  Air  Force’s  Global  Hawk  is  a  high-altitude,  long- 
endurance  unmanned  aircraft  with  integrated  sensors  and  ground 
stations  providing  intelligence,  surveillance,  and  reconnaissance 
capabilities. 

8.  Space  Based  Infrared  System  (SBIRS)  High:  The  Air  Force’s  SBIRS 
High  satellite  system  is  being  developed  to  replace  the  Defense 
Support  Program  and  perform  a  range  of  missile  warning,  missile 
defense,  technical  intelligence,  and  battle-space  awareness  missions. 

GAO  data  were  available  for  the  following  years: 

1.  Excalibur:  2004-201 1  (eight  years) 


1  Called  the  Advanced  Amphibious  Assault  Vehicle  (AAAV)  in  2003. 

2  Also  called  the  JTRS  Ground  Mobile  Radios  (GMR). 

3  Also  called  the  F-35. 

4  Also  called  Global  Hawk  RQ-4A/B. 
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2.  EFV:  2003-2011  (nine  years) 

3.  C-5  RERP:  2004-201 1  (eight  years) 

4.  JTRS:  2004-2011  (eight  years) 

5.  JSF:  2003-201 1  (nine  years) 

6.  Reaper:  2004-201 1  (eight  years) 

7.  Global  Hawk:  2004-201 1  (eight  years) 

8.  SBIRS:  2003-201 1  (nine  years) 

An  example  of  a  GAO  write-up  is  shown  in  Appendix  B.  Key  information  from 
the  GAO  reports  used  in  this  research  project  are  the  following: 

1.  Financial  Data 

■  Research  &  Development  (R&D)  cost  (total  estimated) 

■  R&D  funding  needed  to  complete 

■  Yearly  R&D  cost 

■  Procurement  cost 

■  Procurement  quantity 

■  Unit  cost 

2.  Schedule 

■  Development  start 

■  Low-rate  decision 

■  Full-rate  decision 

■  Last  procurement 

■  IOC  (initial  operating  capability) 

■  Acquisition  time 

■  EDM  (engineering  development  model) 

■  Design  and/or  technical  reviews 

3.  Technical 

■  Technical  maturity 

■  Design  maturity  (primarily  the  percent  of  drawings  released  to 
manufacturing) 
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■  Production  maturity  (primarily  the  use  of  statistical  process 
control) 

■  Software  development 

Of  these  sources  of  information,  the  group  called  Technical  is  directly  related 
to  technical  risk  management.  Other  risk-management  considerations  include  the 
overall  acquisition  strategy,  use  of  prototypes  (including  EDMs  and  LRIP  units),  and 
the  scheduling  of  design  and  technical  reviews. 

Not  all  of  the  write-ups  contained  all  of  this  information.  One  area  that  was 
poorly  documented  was  software  development.  As  an  approximation,  the  number  of 
sentences  in  each  write-up  that  mentioned  software  was  counted  and  used  as  an 
indicator  of  the  importance  of  software  to  the  program.  This  assumption  is  not 
expected  to  be  a  data-driven  measure  of  software  risk. 

The  yearly  R&D  expenditures  were  not  listed  in  the  GAO  reports,  but  they 
were  derived  from  the  yearly  values  for  R&D  costs  and  total  R&D  costs  to  complete. 
That  is,  by  using  each  year’s  “Research  and  development  costs”  from  the  Program 
Performance  table  and  the  “R&D  funding  needed  to  complete”  from  the  Program 
Essentials  table,  it  was  possible  to  determine  the  amount  of  R&D  funds  spent  each 
year. 

All  financial  figures  were  adjusted  for  inflation  and  put  into  201 1  dollars.  The 
yearly  inflation  rates  were  obtained  by  looking  at  the  “As  of ...”  cost  listed  in  the 
Program  Performance  tables  and  backing  out  the  yearly  inflation  rates.  The  C-5 
RERP,  JSF,  and  SBIRS  programs  had  nearly  identical  inflation  rates  and  the  values 
from  the  C-5  RERP  values  were  used  in  this  analysis  (shown  in  Error!  Reference 
source  not  found.).  Use  of  the  inflation  rates  is  important  when  trying  to  assess 
year-to-year  cost  increases  since  costs  were  reported  in  current  dollars  for  the  time 
period  of  each  report. 
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2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 


Figure  1 .  Inflation  Rates  Used  for  the  Financial  Analysis  of  the 

GAO  Cost  Figures 
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Research  Methodology 


A  broad  definition  of  research  was  given  by  Martyn  Shuttleworth  (2008):  “In 
the  broadest  sense  of  the  word,  the  definition  of  research  includes  any  gathering  of 
data,  information  and  facts  for  the  advancement  of  knowledge.” 

Another  definition  of  research  was  given  by  J.W.  Creswell  (2008)  who  stated, 
“Research  is  a  process  of  steps  used  to  collect  and  analyze  information  to  increase 
our  understanding  of  a  topic  or  issue.”  According  to  Creswell,  research  consists  of 
three  steps:  pose  a  question,  collect  data  to  answer  the  question,  and  present  an 
answer  to  the  question. 

Research  can  be  defined  as  the  search  for  knowledge,  or  as  any  systematic 
investigation,  with  an  open  mind,  to  establish  novel  facts,  solve  new  or  existing 
problems,  prove  new  ideas,  or  develop  new  theories,  usually  using  a  scientific 
method.  Scientific  research  relies  on  the  application  of  the  scientific  method.  This 
research  provides  scientific  information  and  theories  for  the  explanation  of  the  nature 
and  the  properties  of  the  world  around  us. 

A.  Research  Strategy 

This  effort  is  Phase  I  of  a  multi-phase  research  project  (Naval  Postgraduate 
School,  BAA  Number  NPS-BAA-1 1-02).  The  hypothesis  of  this  multi-phase  research 
project  is  that  when  risks  increase  during  development,  they  need  to  be  continuously 
reflected  in  cost,  schedule,  and  performance.  This  is  not  a  radically  new  idea.  The 
DoD  and  contractors  treat  risk  as  a  component  of  cost,  schedule,  and  performance 
when  the  government  issues  a  request  for  proposal  (RFP)  and  the  contractor 
submits  a  bid. 

The  overall  research  project  will  be  done  in  phases  with  a  separate  research 
paper  for  each  phase.  This  “incremental”  approach,  which  in  itself  is  a  risk-mitigation 
strategy,  will  allow  the  researcher  to  focus  on  specific  research  questions  and  will 
allow  timelier  release  of  research  reports.  The  overall  research  approach  is  as 
follows: 
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■  Phase  I:  Conduct  a  review  of  the  treatment  of  risk  on  several  major 
weapon  systems  using  GAO  reports  (this  report); 

■  Phase  II:  Conduct  a  survey  of  the  DoD  acquisition  workforce  in  the 
risk-taking  behavior  of  the  workforce  under  different  situations 
(McKeon,  2012a); 

■  Phase  III:  Conduct  experiments  on  the  risk-taking  behavior  of  the 
workforce  under  different  situations  (McKeon,  2012b); 

■  Phase  IV:  Develop  a  stochastic  computer  simulation  to  model  and  test 
the  risk-taking  characteristics  of  government  workforce  members;  and 

■  Phase  V:  Interview  DoD  and  industry  project  managers  (or 
equivalents)  in  best  practices  of  risk  management. 

This  research  paper  covers  Phase  I  of  this  research  project. 

B.  Data  Collection 

For  Phase  I  of  the  overarching  research  project  on  risk  management  within 
the  DoD,  the  yearly  GAO  reports  on  major  DoD  weapon  systems  were  used  in  this 
research  study.5  The  yearly  GAO  reports  are  released  in  March  and  generally 
provide  a  two-page  analysis  of  the  largest  major  defense  programs.  The  reports  are 
public-disclosure  documents  and  include  financial  data  and  technical  maturity 
information.  The  GAO  reports  exist  from  2003-201 1 ,  although  the  first  report  (2003) 
covered  a  smaller  number  of  programs  than  the  reports  published  from  2003-201 1 . 

Analysis  of  the  GAO  reports  consisted  of  examining  the  write-up  for  each 
program  over  the  nine-year  period,  looking  for  trends  in  the  data,  identifying  risk- 
mitigation  activities,  and  evaluating  the  overall  effectiveness  of  the  risk  management 
process. 

C.  Analysis  of  the  GAO  Reports 

The  GAO  reports  were  reviewed  to  obtain  the  following  data: 

1.  Program  Schedule 

a)  There  are  several  values  that  relate  to  schedule.  The  most 
obvious  is  the  “Acquisition  cycle  time  (months)”  that  is  in  the 

5  Personnel  from  the  programs  offices  were  not  interviewed  for  this  study. 
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main  table  on  the  first  page  of  the  report.  This  measures  the 
time  from  “development  start”  to  “initial  capability.” 

b)  Another  schedule  parameter  is  “development  start”  to  “low-rate 
decision.”  This  was  not  reported,  but  it  was  easily  determined 
from  the  schedule  given  on  the  first  page  of  each  write-up.  In 
some  cases,  the  time  to  LRIP  increased  but  the  “acquisition 
time”  or  the  time  to  FRP  did  not  change,  which  suggested  a 
high-risk  approach  to  planning. 

c)  Another  schedule  parameter  was  “development  start”  to  “full- 
rate  decision.”  This  was  an  important  measure  for  programs  that 
entered  LRIP  at  a  low  maturity  level.  The  time  to  FRP  captured 
the  time  to  mature  the  design  and  production  processes. 

2.  Program  Costs.  There  are  several  important  cost  values  listed  on  the  first 
page  of  each  report. 

a)  R&D  costs:  There  is  a  cost  stated  at  the  start  of  development 
and  the  most  current  estimate  at  completion.  The  values  are 
adjusted  for  inflation.  The  current  estimate  at  completion  is  the 
single  most  important  piece  of  information  used  in  this  research 
project. 

b)  Procurement  unit  cost  was  also  included  in  each  GAO  report. 

c)  Program  unit  cost:  This  value  was  given  in  each  report  and  was 
used  in  this  analysis.  The  value  is  dependent  on  the  quantity 
fielded  and  so  changes  in  quantity  have  to  be  considered  when 
evaluating  this  cost  data. 

d)  Funding  needed  to  complete:  R&D.  This  value  was  very 
important  because  it  gave  a  snapshot  of  the  technical  state  of 
the  program.  This  value  and  the  R&D  costs  were  used  to 
determine  the  R&D  dollars  spent  every  year. 

3.  Total  Quantities 

a)  The  total  number  of  systems  planned  to  be  purchased  for 

procurement  was  listed  on  the  first  page  of  each  report.  This 
information  was  important  for  understanding  the  program  and 
procurement  unit  costs. 

4.  Technical  Maturity 
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a)  There  was  a  chart  called  “Attainment  of  Product  Knowledge” 

which  by  itself  was  not  very  useful.  However,  each  report  had  a 
section  called  “Technology  Maturity.”  This  was  very  useful  to 
define  the  technical  maturity  of  the  program,  but  technical 
maturity  rarely  correlated  with  program  success. 

5.  Design  Maturity 

There  was  very  useful  information  in  the  section  called  “Design  Maturity.” 

a)  The  GAO  reports  listed  the  percent  of  design  drawings 
complete.  Some  of  the  reported  values  were  not  very  accurate, 
which  will  be  discussed  in  Chapter  4. 

b)  The  reports  sometimes  mentioned  software  development.  It  was 
very  difficult  to  appraise  the  software  maturity  because  metrics 
were  not  used.  However,  as  a  crude  metric,  the  number  of 
sentences  that  mentioned  “software”  were  counted  and  used  to 
assess  the  maturity  of  the  software  development  effort. 

6.  Production  Maturity. 

a)  There  was  a  section  on  production  maturity,  but  it  did  not  use 
manufacturing  readiness  levels  (MRLs).  The  write-ups  often 
said  that  the  assessors  could  not  assess  production  maturity 
because  statistical  process  controls  were  not  being  used.  In 
some  cases,  other  activities  or  accomplishments  were  looked  at 
to  assess  production  maturity. 
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IV. 


Findings 


This  research  project  is  based  on  written  reports  from  the  Government 
Accountability  Office  (GAO)  on  major  DoD  weapon  systems.  The  GAO  reports  come 
out  in  March  of  every  year  and  generally  provide  a  two-page  analysis  of  the  largest 
major  defense  programs.  The  GAO  reports  exist  from  2003-201 1 ,  although  the  first 
report  (2003)  covered  a  smaller  number  of  programs  than  the  reports  published  from 
2003-201 1 .  GAO  data6  were  available  for  the  following  years: 

1.  Excalibur:  2004-201 1  (eight  years) 

2.  EFV:  2003-2011  (nine  years) 

3.  C-5  RERP:  2004-201 1  (eight  years) 

4.  JTRS:  2004-2011  (eight  years) 

5.  JSF:  2003-201 1  (nine  years) 

6.  Reaper:  2004-201 1  (eight  years) 

7.  Global  Hawk:  2004-201 1  (eight  years) 

8.  SBIRS:  2003-201 1  (nine  years) 

A.  Excalibur:  2004-201 1  (Eight  Years) 

The  Excalibur  Precision  Guided  Extended  Range  Artillery  Projectile  yearly  schedules 
and  milestones  from  1997-201 1  are  shown  in  Error!  Reference  source  not  found.. 
Every  published  schedule  is  shown  as  a  row  in  the  table.  For  example,  in  2004,  the 
LRIP  decision  was  scheduled  for  June  2006,  the  full-rate  production  (FRP)  decision 
was  planned  for  June  2008,  and  IOC  was  planned  for  September  2008.  The  2004 
schedule  did  not  show  a  design  review  in  2005,  but  starting  in  2006  the  schedule 
showed  a  design  review  in  either  April  or  May  of  2005. 


6  Disclaimer:  Some  information  in  the  GAO  reports  was  incomplete  and  some  programs  had 
significant  changes  in  program  scope  that  complicated  the  analyses  of  the  programs. 
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The  milestone  chart  is  useful  because  it  clearly  shows  schedule  slip  in  FRP 
and  IOC  as  a  function  of  time.  For  example,  IOC  was  originally  planned  to  occur  in 
September  2008  and  that  schedule  was  in  place  until  2008  when  IOC  slipped  to 
January  2009.  In  2009,  the  forecasted  IOC  slipped  13  months.  Overall  IOC  slipped 
29  months  in  the  three  years  from  2009-201 1 . 

Error!  Reference  source  not  found,  is  another  way  to  look  at  the  program 
milestones.  Three  curves  are  plotted:  time  to  LRIP,  time  to  FRP,  and  total  acquisition 
cycle  time  (all  measured  in  months  from  the  start  of  program  development).  The  first 
two  datasets  are  derived  from  the  yearly  schedules,  while  the  last  one  is  from  the 
“Program  Performance”  table  in  every  GAO  write-up.  The  schedule  data  looks  very 
favorable  through  2008,  then  starts  to  show  significant  schedule  delays.  The  overall 
program  schedule  delay  was  35  months  in  a  four-year  period. 

Error!  Reference  source  not  found,  shows  the  R&D  costs  for  the  program. 
All  costs  have  been  adjusted  for  inflation.  The  GAO  reports  did  not  report  a  yearly 
R&D  spending,  so  it  was  derived7  from  the  reported  cost  data.  In  the  figure,  the 
planned  total  program  cost  is  shown  as  the  black  line  with  squares.  There  was  a 
large  program  cost  increase  in  2005.  The  “R&D  funding  needed  to  complete”  the 
program,  listed  in  the  “Program  Essentials”  block  in  each  GAO  report,  is  shown  as 
the  green  line  with  triangles. 

The  red  line  with  asterisks  is  the  actual  R&D  spending  for  each  year.  The  red 
curve  clearly  shows  the  drop  in  R&D  funding  from  2007  to  2008.  Spending  was  flat 
to  2007  and  then  it  was  flat,  but  at  a  lower  level,  until  2010.  The  blue  line  with 
diamonds  is  total  R&D  spending  from  the  beginning  of  the  program. 


7  The  R&D  spending  for  a  year  was  not  reported,  but  it  could  be  determined  from  the  reported  R&D 
costs  from  two  years  of  data.  Mathematically,  R&D  spent  (year  /')  =  (Budgeted(year  /')  - 
Budgeted(year  /  —  1 ))  —  (Needed  to  Complete  (year  /')  -  Needed  to  complete^'  -  1 )). 
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Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs8  and  the  program  unit  costs.9  There  are  large  quantity 
changes  in  2005,  2006,  and  201 1 .  The  procurement  and  program  unit  costs 
increased  significantly  in  201 1 .  The  program  unit  cost  increased  by  193%  and  the 
procurement  cost  increased  by  93%.  There  was  a  Nunn-McCurdy10  breach  in  2010 
when  the  production  quantity  dropped  and  the  procurement  unit  cost  increased  from 
$47,000  to  $99,000. 

Error!  Reference  source  not  found,  shows  the  technical  maturity  for  each 
year  based  on  the  GAO  reports.  Three  critical  technologies  were  identified  for  the 
program  and  it  was  reported  that  all  technologies  were  mature  starting  in  2006. 

Error!  Reference  source  not  found,  shows  the  percent  of  drawings 
completed.  It  shows  a  flat  trend  at  100%  from  2006  until  2010  and  then  a  decrease 
to  90%  in  2011.  The  percent  drawings  completed  is  not  a  good  measure  of  design 
maturity  because  $257  million  (est.)  was  spent  on  R&D  from  2007  until  2010,  even 
though  it  was  reported  that  100%  of  the  design  drawings  had  been  completed. 
Design  problems  were  not  recognized  until  2011. 

None  of  the  yearly  Excalibur  write-ups  mentioned  software.  There  is 
undoubtedly  software  or  firmware  being  developed  or  modified,  such  as  system 


8  The  procurement  unit  cost  is  the  total  procurement  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  Procurement  unit  cost  does  not 
include  R&D  costs. 

9  The  program  unit  cost  is  the  total  program  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  The  total  program  costs  are  the 
R&D  costs  plus  the  procurement  costs. 

10  The  Nunn-McCurdy  Amendment  or  Nunn-McCurdy  Provision,  introduced  by  Senator  Sam  Nunn 
and  Congressman  Dave  McCurdy  in  the  United  States  1982  Defense  Authorization  Act,  is  designed 
to  curtail  cost  growth  in  American  weapons  procurement  programs.  It  requires  notification  of  the 
United  States  Congress  if  the  cost  per  unit  grows  more  than  15%  beyond  what  was  originally 
estimated,  and  calls  for  the  termination  of  programs  with  total  cost  growth  greater  than  25%,  unless 
the  Secretary  of  Defense  submits  a  detailed  explanation  certifying  1)  the  program  is  essential  to 
national  security,  and  that  no  suitable  alternative  of  lesser  cost  is  available;  2)  new  estimates  of  total 
program  costs  are  reasonable;  and  3)  management  structure  is  (or  has  been  made)  adequate  to 
control  costs. 
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operation,  operation,  targeting,  and  so  forth.  The  GAO  reports  do  not  indicate  that 
software  was  an  issue  for  the  program. 


Project 
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•ReX 

•DR 
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FRP*  *IOC 
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•ReX 
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FRP  **100 
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FRP* 

•IOC 

2009 
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DR  ••LRIP 

FRP*  *IOC 
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•DR 
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Figure  2.  Excalibur:  Milestones  Based  on  GAO  Reports 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  ReX  (restart)  are  shown. 
The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells  are  planned  to 
occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred11  at  the  time  of 
the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at  the  time  of  the 
report  for  the  years  after  the  GAO  report. 


Figure  3.  Excalibur  Milestone  Schedule 


11  In  theory,  the  milestones  in  green  should  not  change  from  year  to  year;  however,  in  some  cases 
there  are  minor  differences. 
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Note.  The  red  line  with  triangles  is  the  program  acquisition  cycle  time  listed  in  the  GAO  write-ups  (in 
months).  The  blue  line  with  diamonds  is  the  time  from  development  start  to  LRIP  (in  months).  The 
black  line  with  boxes  is  the  time  from  development  start  to  FRP  (in  months). 


Year 


—  —  Plan  to  spend  until  completion 
HIH  Planned  total  cost 

M  Actual  spent  in  year 

—  —  Actual  Total  Spent 


Figure  4.  Excalibur  R&D  Cost  Profile 

Note.  The  green  line  with  triangles  is  the  “R&D  funding  needed  to  complete”  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  black  line  with  boxes  is  the  planned  total 
R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  The  red  line  with  black 
asterisks  is  the  R&D  spending  during  the  year.  The  blue  line  with  diamonds  is  the  Total  R&D 
spending  from  the  beginning  of  the  project.  It  is  derived  from  the  “funding  needed  to  complete”  and 
the  “R&D  cost”  categories  from  the  reports. 
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$0,250  -  QtY 


Figure  5.  Excalibur:  Production  Quantity  and  Unit  Costs 

Note.  The  black  line  with  squares  is  the  procurement  unit  cost  adjusted  for  inflation.  The  red  line  with 
triangles  is  the  program  unit  cost  adjusted  for  inflation  (the  program  unit  cost  includes  the  R&D  costs 
amortized  over  the  total  production  lot). 


Figure  6.  Excalibur:  Technical  Maturity 
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Figure  7.  Excalibur:  Drawings  Completed 

B.  Expeditionary  Fighting  Vehicle  (EFV):  2003-2011 

The  EFV  (formerly  the  AAAV)  program  was  in  the  GAO  reports  from  2003- 
2011.  The  program  was  cancelled  in  201 1  after  the  GAO  report  was  issued.  Even 
though  the  program  has  been  terminated,  the  past  data  is  useful  for  understanding 
program  risks. 

The  schedules  for  the  period  2001-201 1  are  shown  in  Error!  Reference 
source  not  found..  Every  published  schedule  is  shown  as  a  row  in  the  table.  The 
columns  represent  the  different  years  in  each  yearly  schedule.  For  example,  in  2003 
the  LRIP  decision  was  scheduled  for  September  2005,  the  full-rate  production 
decision  was  planned  for  August  2007,  and  IOC  was  planned  for  September  2008. 
This  chart  is  useful  because  it  clearly  shows  schedule  slip  as  a  function  of  time. 
There  was  a  Nunn-McCurdy  breach  in  2007,  which  shows  up  as  a  large  schedule 
slip  between  2007  and  2008. 

Error!  Reference  source  not  found,  is  another  way  to  look  at  the  program 
milestones.  Three  curves  are  plotted:  time  to  LRIP,  time  to  FRP,  and  total  acquisition 
cycle  time  (all  measured  in  months  from  the  start  of  program  development).  The  first 
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two  datasets  are  derived  from  the  yearly  schedules,  while  the  last  one  is  from  the 
“Program  Performance”  table  in  every  GAO  write-up.  The  chart  shows  the  large 
increase  in  schedule  in  2008. 

Error!  Reference  source  not  found,  shows  the  R&D  costs  for  the  program. 
All  costs  have  been  adjusted  for  inflation.  The  GAO  reports  did  not  report  a  yearly 
R&D  spending,  so  it  was  derived  from  the  reported  cost  data.  In  the  figure,  the 
planned  total  program  cost  is  shown  as  the  black  line  with  squares.  The  “R&D 
funding  needed  to  complete”  the  program,  listed  in  the  “Program  Essentials”  block  in 
each  GAO  report,  is  shown  as  the  green  line  with  triangles. 

The  red  line  with  asterisks  is  the  actual  R&D  spending  for  each  year. 

Spending  was  flat  throughout  the  program.  The  blue  line  with  diamonds  is  the  actual 
R&D  spending  from  the  beginning  of  the  program. 

The  “planned  total  cost”  (black  curve)  shows  a  step  increase  every  other  year, 
which  may  be  an  artifact  of  the  data  collection  or  budget  reporting  and  is  not  a 
change  in  scope  of  the  program.  There  was  a  steady  increase  in  planned  total  cost 
throughout  the  life  of  the  program,  with  a  large  step  increase  in  2008  due  to  the 
Nunn-McCurdy  re-baseline.  It  is  interesting  to  note  that  the  cost  increases  were 
about  equal  to  the  expenditures,  so  the  program  was  not  getting  closer  to 
completion. 

Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs,12  and  the  program  unit  costs.13  There  was  a  large  quantity 
change  in  2008  (a  42%  decrease)  and  large  increases  in  the  procurement  and 
program  unit  costs.  The  program  unit  cost  increased  by  107%  and  the  procurement 
cost  increased  by  88%. 


12  The  procurement  unit  cost  is  the  total  procurement  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  Procurement  unit  cost  does  not 
include  R&D  costs. 

13  The  program  unit  cost  is  the  total  program  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  The  total  program  costs  are  the 
R&D  costs  plus  the  procurement  costs. 
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Error!  Reference  source  not  found,  shows  the  technical  maturity  for  each 
year  based  on  the  GAO  reports.  Initially,  the  program  reported  five  critical 
technologies,  but  then  it  dropped  to  four  starting  in  2008.  It  was  reported  that  all 
technologies  were  mature  starting  in  2005. 

Error!  Reference  source  not  found,  shows  the  number  of  design  drawings 
completed  for  each  year.  The  data  show  that  the  program  did  not  have  a  good 
handle  on  the  quality  and/or  number  of  drawings  completed.  The  data  showed  that 
100%  of  the  drawings  had  been  completed  in  2005,  although  R&D  spending  was  still 
on  the  order  of  $300  million,  which  implies  that  some  redesign  of  critical  components 
was  underway.  Starting  in  2006,  there  was  a  recognition  that  redesign  was 
necessary  although  the  magnitude  of  the  redesign  effort  was  underestimated. 

The  GAO  write-ups  do  not  have  a  specific  measure  of  software  maturity.  As 
an  approximation,  the  number  of  sentences  in  each  report  that  mentioned  software 
was  counted  as  a  gauge  of  the  software  development.  Error!  Reference  source 
not  found,  shows  the  number  of  lines  in  the  EFV  GAO  reports  that  mention 
software.  The  GAO  write-ups  mentioned  software  10  times  in  2007,  but  software 
wasn’t  mentioned  from  2008-2010. 
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Figure  8.  EFV  Schedule 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  Dev  Start  (development 
start)  are  shown.  The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells 
are  planned  to  occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred 
at  the  time  of  the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at 
the  time  of  the  report  for  the  years  after  the  GAO  report. 
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Figure  9.  EFV  Schedule  (Months  From  Development  Start) 

Note.  The  red  line  with  triangles  is  the  program  acquisition  cycle  time  listed  in  the  GAO  write-ups  (in 
months).  The  blue  line  with  diamonds  is  the  time  from  development  start  to  LRIP  (in  months).  The 
black  line  with  boxes  is  the  time  from  development  start  to  FRP  (in  months).  When  data  points  are 
missing,  a  dashed  line  is  drawn  between  points.  The  dashed  lines  are  for  visual  purposes  only. 
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Figure  10.  EFV  R&D  Cost  Profile 

Note.  The  green  line  with  triangles  is  the  “R&D  funding  needed  to  complete”  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  black  line  with  boxes  is  the  planned  total 
R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  The  red  line  with  black 
asterisks  is  the  R&D  spending  during  the  year.  The  blue  line  with  diamonds  is  the  Total  R&D 
spending  from  the  beginning  of  the  project.  It  is  derived  from  the  “funding  needed  to  complete”  and 
the  “R&D  cost”  categories  in  the  reports. 
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Figure  11.  EFV  Production  Plan  and  Unit  Cost 

Note.  The  black  line  with  squares  is  the  procurement  unit  cost  adjusted  for  inflation.  The  red  line  with 
triangles  is  the  program  unit  cost  adjusted  for  inflation  (the  program  unit  cost  includes  the  R&D  costs 
amortized  over  the  total  production  lot). 


Figure  12.  EFV:  Technical  Maturity 
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Figure  13.  EFV:  Design  Maturity  Based  on  the  Number  of  Drawings  Completed 


Figure  14.  EFV:  Number  of  Lines  in  the  GAO  Reports  That 

Mentioned  Software 

C.  C-5  Reliability  Enhancement  and  Reengining  Program 
(RERP):  2004-2011  (Eight  Years) 

Error!  Reference  source  not  found,  shows  the  published  C-5  RERP  yearly 
schedules  and  milestones  from  2004-201 1 .  Every  published  schedule  is  shown  as  a 
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row  in  the  table.  The  columns  represent  the  different  years  in  each  yearly  schedule. 
The  yearly  schedules  are  very  misleading.  Looking  at  the  data,  only  year  2008 
shows  a  big  change  in  schedule.  There  was  a  Nunn-McCurdy  breach  in  2008. 

Error!  Reference  source  not  found,  is  another  way  to  look  at  the  program 
milestones.  Three  curves  are  plotted:  time  to  LRIP,  time  to  IOC,  and  total  acquisition 
cycle  time  (all  measured  in  months  from  the  start  of  program  development).  The  first 
two  datasets  are  derived  from  the  yearly  schedules,  while  the  last  one  is  from  the 
“Program  Performance”  table  in  every  GAO  write-up.  The  chart  shows  the  large 
increase  in  schedule  in  2008. 

Error!  Reference  source  not  found,  shows  the  R&D  costs  for  the  program. 
All  costs  have  been  adjusted  for  inflation.  The  GAO  reports  did  not  report  a  yearly 
R&D  spending,  so  it  was  derived  from  the  reported  cost  data.  In  the  figure,  the 
planned  total  program  cost  is  shown  as  the  black  line  with  squares.  The  reason(s) 
for  the  drop  in  “planned  total  R&D  costs”  in  2006  and  2007  is/are  unknown,  but  may 
be  the  result  of  a  mathematical  error  in  the  reported  data. 

The  “R&D  funding  needed  to  complete”  the  program,  listed  in  the  “Program 
Essentials”  block  in  each  GAO  report,  is  shown  as  the  green  line  with  triangles.  The 
red  line  with  asterisks  is  the  actual  R&D  spending  for  each  year.  R&D  spending  was 
dropping  every  year.  The  blue  line  with  diamonds  is  the  actual  R&D  spending  from 
the  beginning  of  the  program. 

Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs,14  and  the  program  unit  costs.15  The  quantity  dropped  from 
111  to  52  in  2009  (a  53%  decrease).  Unit  costs  (including  R&D)  rose  from  $87 
million  to  $137  million  (a  57%  increase).  (The  quantity  and,  therefore,  the 


14  The  procurement  unit  cost  is  the  total  procurement  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  This  cost  does  not  include  R&D 
costs. 

15  The  program  unit  cost  is  the  total  program  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  The  total  program  costs  are  the 
R&D  costs  plus  the  procurement  costs. 
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procurement  unit  cost  in  2008  seem  to  be  in  error.  The  quantity  for  2008  should 
have  been  52.) 

Error!  Reference  source  not  found,  shows  the  technical  maturity  for  each 
year  based  on  the  GAO  reports.  One  critical  technology  was  reported  for  the 
program  (the  new  engine)  and  since  it  was  a  commercially  available  engine,  the 
program  reported  it  as  mature  throughout  the  program. 

In  many  cases,  design  maturity  was  based  on  the  number  of  drawings 
released.  There  are  a  couple  of  points  to  consider.  First,  many  of  the  quoted  values 
seem  to  be  arbitrary.  In  Error!  Reference  source  not  found.,  the 
program/contractor  reported  98%  for  three  years.  On  might  ask  if  they  stopped 
working  on  drawings  in  2003?  Then  the  number  of  drawings  dropped  to  90%,  then 
80%,  then  back  to  90%,  and  then  to  100%.  Nothing  works  that  predictably.  The 
“number  of  drawings  completed”  that  were  reported  do  not  seem  realistic. 

The  GAO  write-ups  do  not  have  a  specific  measure  of  software  maturity.  As 
an  approximation,  the  number  of  sentences  in  each  report  that  mentioned  software 
was  counted  as  a  gauge  of  the  software  development.  Error!  Reference  source 
not  found,  shows  the  number  of  lines  in  the  GAO  reports  that  mention  software. 
Software  was  mentioned  15  times  between  2004  and  2006,  but  only  once  between 
2007  and  201 1 .  In  2004,  the  write-up  mentioned,  “According  to  program  officials,  the 
greatest  risk  ...  is  software  development  and  integration  activities.” 
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Figure  15.  C-5  RERP  Yearly  Schedule 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  ReX  (restart)  are  shown. 
The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells  are  planned  to 
occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred  at  the  time  of 
the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at  the  time  of  the 
report  for  the  years  after  the  GAO  report. 


Figure  16.  C-5  RERP  Schedule  (Months  From  Program  Start) 

Note.  The  red  line  with  triangles  is  the  program  acquisition  cycle  time  listed  in  the  GAO  write-ups  (in 
months).  The  blue  line  with  diamonds  is  the  time  from  development  start  to  LRIP  (in  months).  The 
black  line  with  boxes  is  the  time  from  development  start  to  FRP  (in  months).  When  data  points  are 
missing,  a  dashed  line  is  drawn  between  points.  The  dashed  lines  are  for  visual  purposes  only. 
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Figure  17.  C-5  RERP:  R&D  Costs 

Note.  The  green  line  with  triangles  is  the  “R&D  funding  needed  to  complete”  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  black  line  with  boxes  is  the  planned  total 
R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  The  red  line  with  black 
asterisks  is  the  R&D  spending  during  the  year.  The  blue  line  with  diamonds  is  the  Total  R&D 
spending  from  the  beginning  of  the  project.  It  is  derived  from  the  “funding  needed  to  complete”  and 
the  “R&D  cost”  categories  from  the  reports. 
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Figure  18.  C-5  RERP:  Total  Production  and  Unit  Cost 

Note.  The  black  line  with  squares  is  the  procurement  unit  cost  adjusted  for  inflation.  The  red  line  with 
triangles  is  the  program  unit  cost  adjusted  for  inflation  (the  program  unit  cost  includes  the  R&D  costs 
amortized  over  the  total  production  lot). 

The  quantity  for  2008  should  have  been  52. 


Figure  19.  C-5  RERP:  Technical  Maturity 
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Figure  20.  C-5  RERP:  Design  Drawings  Completed 


Figure  21.  C-5  RERP:  Number  of  Lines  Mentioning  Software 


D.  Joint  Tactical  Radio  System  (JTRS):  2004-2011  (Eight 
Years) 

The  JTRS  program  scope  changed  during  the  evaluation  period.  This  is  one 
program  where  software  was  mentioned  repeatedly  because  it  is  a  critical 
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component  of  the  JTRS  system.  Error!  Reference  source  not  found,  shows  the 
program  milestones  that  were  published  each  year  in  the  GAO  reports.  This  chart  is 
useful  because  it  clearly  shows  schedule  slip  as  a  function  of  time.  Every  published 
schedule  is  shown  as  a  row  in  the  table.  The  columns  represent  the  different  years 
in  each  yearly  schedule.  For  example,  in  2005,  the  LRIP  decision  was  scheduled  for 
April  2006  and  the  full-rate  production  decision  was  planned  for  June  2007. 

The  2006  GAO  write-up  had  the  following  information  regarding  the  program 
structure:  “The  JTRS  Cluster  1  program  is  currently  being  restructured  due  to 
significant  cost  and  schedule  problems  that  came  to  light  in  late  2004.”  Milestones 
were  not  given  in  the  2006  GAO  report;  however,  they  were  given  in  the  2007  report. 
From  2005  to  201 1 ,  LRIP  slipped  four  years  and  five  months  and  the  FRP  decision 
slipped  five  years  and  five  months. 

Error!  Reference  source  not  found,  shows  the  schedule  based  on  the 
schedule  listed  in  the  GAO  reports  (red  line),  the  schedule  until  LRIP  (blue  line)  and 
the  schedule  to  FRP  (black  line).  Values  are  measured  from  the  start  of  program 
development.  Data  were  not  reported  for  2006  due  to  the  restructuring.  The  time  to 
LRIP  and  the  time  to  FRP  essentially  doubled  between  2005  and  2007  (a  100% 
increase  in  the  schedule). 

Error!  Reference  source  not  found,  shows  the  R&D  costs  for  the  program. 
All  costs  have  been  adjusted  for  inflation.  The  GAO  reports  did  not  report  a  yearly 
R&D  spending,  so  it  was  derived  from  the  reported  cost  data.  In  the  figure,  the 
planned  total  program  costs  are  shown  as  the  black  line  with  squares.  The  reported 
value  for  2009  was  artificially  low  and  it  was  increased  up  by  $172  million  so  that  a 
realistic  program  total  cost  was  given  for  2009.  The  data  show  big  total  program  cost 
increases  for  2006  and  2007  (a  total  of  73%  from  2005-2007). 

The  R&D  funding  needed  to  complete  the  program,  listed  in  the  “Program 
Essentials”  block  in  each  GAO  report,  is  shown  as  the  green  line  with  triangles.  In 
the  2010  GAO  report,  the  “R&D  funding  needed  to  complete”  data  point  was 
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missing.  A  value  of  $220  million  was  assumed  because  it  resulted  in  a  linear 
spending  profile  from  2009-201 1 . 

The  red  line  with  asterisks  is  the  actual  R&D  spending  for  each  year. 
Spending  was  flat  throughout  the  program.  The  blue  line  with  diamonds  is  the  actual 
R&D  spending  from  the  beginning  of  the  program. 

Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs,16  and  the  program  unit  costs.17  There  was  a  large  decrease 
in  quantity  in  2009  (a  17%  decrease)  and  large  increases  in  the  procurement  and 
program  unit  costs  (25%). 

Error!  Reference  source  not  found,  shows  the  technical  maturity  for  each 
year  based  on  the  GAO  reports.  Twenty  critical  technologies  were  identified  for  the 
program.  Except  for  2009,  which  seems  to  be  in  error,  the  highest  percentage  of 
technologies  that  were  reported  as  mature  was  65%. 

On  most  programs,  the  GAO  uses  the  number  of  released  drawings  to 
measure  design  maturity.  Values  were  only  provided  for  three  years  (see  Figure 
27).  The  program  reported  100%  in  2005,  but  83%  in  2007  and  2008. 

The  GAO  write-ups  do  not  have  a  specific  measure  of  software  maturity.  As 
an  approximation,  the  number  of  sentences  in  each  report  mentioning  software  was 
counted  as  a  gauge  of  the  software  development.  Error!  Reference  source  not 
found,  shows  the  number  of  lines  in  the  GAO  reports  that  mentioned  software. 

While  the  write-ups  discussed  the  software  issues,  the  references  to  software  were 
not  proportional  to  the  software  problems  the  program  faced. 


16  The  procurement  unit  cost  is  the  total  procurement  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  Procurement  unit  cost  does  not 
include  R&D  costs. 

17  The  program  unit  cost  is  the  total  program  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  The  total  program  costs  are  the 
R&D  costs  plus  the  procurement  costs. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-35- 


Project 

1997 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

2010 

2011 

2012 

JTRS  ClustQl|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|(34 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

Ql|Q2|Q3|Q4 

1997 

2000 

•Prog  Start 

2001 

•Prog  Start 

2002 

•Prog  Start 

•Dev  Start 

2003 

•Prog  Start 

•Dev  Start 

2004 

•Prog  Start 

•DevStart 

DR 

•FRP 

2005 

•Prog  Start 

•Dev  Start  I 

•LRIP 

•FRP 

2006 

•Prog  Start 

•Dev  Start 

DR 

2007 

•Prog  Start 

•Dev  Start 

•DR 

•LRIP 

•F 

2008 

•Prog  Start 

•Dev  Start 

•DR 

•LRIP 

FRP 

•IOC 

2009 

•Prog  Start 

•Dev  Start 

DR* 

•Restart 

•LRIP 

•IOC 

2010 

•Prog  Start 

•Dev  Start 

•Restart 

•LRIP 

•1 

2011 

•Prog  Start 

•Dev  Start 

DR* 

•Restart 

•LRIP 

Figure  22.  JTRS:  Overall  Schedule 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  DevStart  (development 
start)  are  shown.  The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells 
are  planned  to  occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred 
at  the  time  of  the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at 
the  time  of  the  report  for  the  years  after  the  GAO  report. 


Figure  23.  JTRS:  Schedule  to  LRIP  and  FRP 

Note.  The  red  line  with  triangles  is  the  program  acquisition  cycle  time  listed  in  the  GAO  write-ups  (in 
months).  The  blue  line  with  diamonds  is  the  time  from  development  start  to  LRIP  (in  months).  The 
black  line  with  boxes  is  the  time  from  development  start  to  FRP  (in  months).  When  data  points  are 
missing,  a  dashed  line  is  drawn  between  points.  The  dashed  lines  are  for  visual  purposes  only. 
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Figure  24.  JTRS:  Costs  by  Year  for  the  JTRS  Program 

Note.  The  green  line  with  triangles  is  the  “R&D  funding  needed  to  complete”  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  value  for  201 0  was  missing  and  a  value  of 
$220  million  was  used  to  achieve  a  linear  spending  profile  from  2009-201 1 .  The  black  line  with  boxes 
is  the  planned  total  R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  For 
2009,  the  reported  value  was  revised  up  by  $172  million  so  that  a  realistic  program  total  cost  was 
given  for  2009.  The  red  line  with  black  asterisks  is  the  R&D  spending  during  the  year.  The  blue  line 
with  diamonds  is  the  Total  R&D  spending  from  the  beginning  of  the  project.  It  is  derived  from  the 
“funding  needed  to  complete”  and  the  “R&D  cost”  categories  from  the  reports. 
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Figure  25.  JTRS:  Production  Quantity  and  Unit  Cost 

Note.  The  black  line  with  squares  is  the  procurement  unit  cost  adjusted  for  inflation. 

The  red  line  with  triangles  is  the  program  unit  cost  adjusted  for  inflation  (the  program  unit  cost 
includes  the  R&D  costs  amortized  over  the  total  production  lot). 


Figure  26.  JTRS:  Technical  Maturity 
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Figure  27.  JTRS:  Drawings  Complete 

Note.  Data  were  not  available  for  2006  and  2009-201 1 . 


Figure  28.  JTRS:  Lines  in  the  GAO  Reports  Mentioning  Software 


E.  Joint  Strike  Fighter  (JSF):  2003-201 1  (Nine  Years) 

Error!  Reference  source  not  found,  shows  the  program  milestones  that 
were  published  each  year  in  the  GAO  reports.  Every  published  schedule  is  shown  as 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-39- 


a  row  in  the  table.  The  columns  represent  the  different  years  in  each  yearly 
schedule.  For  example,  in  2004,  the  LRIP  decision  was  scheduled  for  April  2006  and 
IOC  was  planned  for  April  2010  (USMC  version). 

The  “Acquisition  cycle  time”  is  reported  in  every  GAO  report.  Two  values  are 
given:  the  value  given  at  program  start  (an  historical  value)  and  the  most  current 
estimate,  which  might  change  every  year.  In  most  programs,  the  planned  acquisition 
cycle  time  at  program  start  does  not  change  because  it  is  a  historical  value.  For  the 
JSF,  the  “approved”  acquisition  cycle  time  in  the  2003  report  was  quoted  as  185 
months  as  of  October  2001 .  However,  in  the  2010  GAO  report,  the  planned 
acquisition  cycle  time  was  quoted  as  1 16  months  as  of  October  2001 .  That  is,  the 
historical  value  changed.  There  wasn’t  an  explanation  for  why  a  different  value  was 
reported.  The  change  in  acquisition  cycle  times  is  reflected  in  Error!  Reference 
source  not  found.. 

Error!  Reference  source  not  found,  shows  the  time  to  LRIP  and  the  time  to 
IOC  from  the  GAO  reports  (measured  from  development  start).  There  were  very 
large  changes  in  the  “Acquisition  cycle  time”  in  2009  and  201 1 . 

All  costs  have  been  adjusted  for  inflation.  The  GAO  reports  did  not  report  the 
amount  of  R&D  money  spent  each  year,  so  it  was  derived  from  the  reported  cost 
data.  The  RDT&E  cost  data  is  shown  Error!  Reference  source  not  found..  The 
planned  total  costs  had  significant  increases  in  2004  (10%),  2005  (19%),  and  201 1 
(11%). 

Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs,  and  the  program  unit  costs.  There  was  little  change  in  the 
production  quantity  from  2004  until  2011.  Program  and  procurement  unit  costs 
increased  in  2005  (19%),  2007  (~6%),  2008  (5%),  and  201 1  (13%). 

Error!  Reference  source  not  found,  shows  the  technical  maturity  for  each 
year  based  on  the  GAO  reports.  Eight  critical  technologies  were  identified  for  the 
program  and  only  five  were  reported  mature  in  2011. 
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The  design  maturity,  defined  as  the  “percent  of  drawings  completed”,  reached 
99%  in  2008,  but  fell  to  90%  in  2009  (see  Figure  34).  It  is  interesting  to  note  that  the 
percent  drawings  complete  is  a  predictor  of  future  cost  over-runs.  In  201 1 ,  a  13% 
increase  in  RDT&E  costs  was  realized.  When  the  “percent  of  drawings  completed”  is 
an  exact  multiple  of  10%,  the  validity  of  the  value  comes  into  question. 


The  GAO  write-up  in  2006  mentioned  software  in  eight  sentences  out  of  33 
sentences  describing  the  program  (see  Figure  35).  The  report  also  had  the  following: 
“Officials  consider  software  a  high  risk  item.”  However,  in  the  following  years,  there 
was  much  less  emphasis  on  software  (an  average  of  about  two  lines  per  write-up). 
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Figure  29.  JSF  Milestones  As  Reported  in  the  GAO  Major  Weapon 

Systems  Report 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  DevStart  (development 
start)  are  shown.  The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells 
are  planned  to  occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred 
at  the  time  of  the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at 
the  time  of  the  report  for  the  years  after  the  GAO  report. 
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Figure  30.  JSF  Schedule 

Note.  The  red  line  with  triangles  is  the  program  acquisition  cycle  time  listed  in  the  GAO  write-ups  (in 
months).  The  blue  line  with  diamonds  is  the  time  from  development  start  to  LRIP  (in  months).  The 
black  line  with  boxes  is  the  time  from  development  start  to  FRP  (in  months). 
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—Plan  to  spend  until  completion 
HIHPIanned  total  cost 
1  *  Actual  spent  in  year 
— 0—  Actual  Total  Spent 


Figure  31.  JSF:  R&D  Costs 

Note.  The  green  line  with  triangles  is  the  R&D  funding  needed  to  complete  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  black  line  with  boxes  is  the  planned  total 
R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  The  red  line  with  black 
asterisks  is  the  R&D  spending  during  the  year.  The  blue  line  with  diamonds  is  the  Total  R&D 
spending  from  the  beginning  of  the  project.  It  is  derived  from  the  “funding  needed  to  complete”  and 
the  “R&D  cost”  categories  from  the  reports. 
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Figure  32.  JSF  Production  Quantity  and  Unit  Cost 

Note.  The  black  line  with  squares  is  the  procurement  unit  cost  adjusted  for  inflation.  The  red  line  with 
triangles  is  the  program  unit  cost  adjusted  for  inflation  (the  program  unit  cost  includes  the  R&D  costs 
amortized  over  the  total  production  lot). 


Figure  33.  JSF:  Technical  Maturity 
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Figure  34.  JSF:  Design  Maturity 


Figure  35.  JSF:  Number  of  Sentences  in  the  GAO  Write-Ups  That 

Mentioned  Software 


F.  Reaper:  2004-201 1  (Eight  Years) 

The  Reaper  program  scope  changed  during  the  study.  Error!  Reference 
source  not  found,  shows  the  major  milestones  identified  in  the  GAO  reports.  Every 
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published  schedule  is  shown  as  a  row  in  the  table.  The  columns  represent  the 
different  years  in  each  yearly  schedule.  For  example,  in  2004  the  LRIP  decision  was 
scheduled  for  January  2006,  the  full-rate  production  decision  was  planned  for 
January  2008,  and  IOC  was  planned  for  December  2009. 

Error!  Reference  source  not  found,  shows  the  time  to  LRIP  and  the  time  to 
FRP  from  the  GAO  reports.  There  is  a  two-year  slip  in  LRIP  in  the  four  years  from 
2004  to  2008,  and  there  is  about  an  equal  delay  in  the  FRP  decision.  IOC  did  not 
change  from  2004  to  2007,  even  though  LRIP  and  the  FRP  decisions  slipped 
significantly  during  this  time  period. 

Error!  Reference  source  not  found,  shows  the  R&D  costs  for  the  program. 
All  costs  have  been  adjusted  for  inflation.  The  GAO  reports  did  not  report  a  yearly 
R&D  spending,  so  it  was  derived  from  the  reported  cost  data.  In  the  figure,  the 
planned  total  program  cost  is  shown  as  the  black  line  with  squares,  and  it  increases 
rapidly  starting  in  2008  due  to  the  increase  in  production  units. 

The  R&D  funding  needed  to  complete  the  program,  listed  in  the  “Program 
Essentials”  block  in  each  GAO  report,  is  shown  as  the  green  line  with  triangles.  It 
grew  over  time  due  to  the  large  increase  in  production  units.  The  2010  GAO  report 
did  not  list  a  figure  for  this  curve  so  $350  million  was  assumed. 

Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs,18  and  the  program  unit  costs.19  There  were  large  quantity 
changes  in  2008,  2009,  2010,  and  201 1  (from  63  in  2007  to  391  in  201 1 ).  Normally 
the  unit  costs  go  down  as  quantity  increases.  They  went  up  in  2008  and  2010,  but 
went  down  in  2009  and  2011.  From  2007  to  201 1 ,  unit  procurement  cost  increased 
by  170%. 


18  The  procurement  unit  cost  is  the  total  procurement  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  Procurement  unit  cost  does  not 
include  R&D  costs. 

19  The  program  unit  cost  is  the  total  program  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  The  total  program  costs  are  the 
R&D  costs  plus  the  procurement  costs. 
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Error!  Reference  source  not  found,  shows  the  technical  maturity  for  each 
year  based  on  the  GAO  reports.  Four  critical  technologies  were  identified  for  the 
program,  and  it  was  reported  that  all  technologies  were  mature  starting  in  2008. 

The  GAO  based  design  maturity  on  the  number  of  drawings  released.  Error! 
Reference  source  not  found,  shows  the  number  of  design  drawings  completed  for 
each  year.  In  2005  the  “number  of  drawings  completed”  was  35%.  It  increased  to 
85%  in  2005,  then  was  80%  from  2007-2008.  The  GAO  reports  stated  95%  in  2009 
and  100%  in  201 1  (a  value  was  not  given  for  2010). 

The  write-ups  do  not  have  a  specific  measure  of  software  maturity.  As  an 
approximation,  the  number  of  sentences  in  each  report  mentioning  software  was 
counted  as  a  gauge  of  the  software  development.  Error!  Reference  source  not 
found,  shows  the  number  of  lines  in  the  GAO  reports  that  mention  software.  The 
GAO  write-ups  do  not  mention  software  until  2007.  From  2007-2009,  software  was 
mentioned  once  per  year.  It  wasn’t  mentioned  in  2010,  but  it  was  referenced  three 
times  in  201 1 . 
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Figure  36.  Reaper  Schedule 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  DevStart  (development 
start)  are  shown.  The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells 
are  planned  to  occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred 
at  the  time  of  the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at 
the  time  of  the  report  for  the  years  after  the  GAO  report. 
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Figure  37.  Reaper:  Time  to  LRIP  and  Time  to  IOC 

Note.  The  red  line  with  triangles  is  the  program  acquisition  cycle  time  listed  in  the  GAO  write-ups  (in 
months).  The  blue  line  with  diamonds  is  the  time  from  development  start  to  LRIP  (in  months).  The 
black  line  with  boxes  is  the  time  from  development  start  to  FRP  (in  months).  When  data  points  are 
missing,  a  dashed  line  is  drawn  between  points.  The  dashed  lines  are  for  visual  purposes  only. 


Figure  38.  Reaper  R&D  Costs 

Note.  The  green  line  with  triangles  is  the  R&D  funding  needed  to  complete  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  black  line  with  boxes  is  the  planned  total 
R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  The  red  line  with  black 
asterisks  is  the  R&D  spending  during  the  year.  The  blue  line  with  diamonds  is  the  Total  R&D 
spending  from  the  beginning  of  the  project.  It  is  derived  from  the  “funding  needed  to  complete”  and 
the  “R&D  cost”  categories  in  the  reports. 
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Figure  39.  Reaper:  Production  Quantity  and  Unit  Cost 

Note.  The  blue  line  with  diamonds  is  the  total  production  plan  by  year.  Values  are  on  the  left  axis.  The 
red  line  with  triangles  is  the  unit  cost  adjusted  for  inflation  (the  unit  cost  includes  the  R&D  costs 
amortized  over  the  total  production  lot).  Values  are  shown  on  the  right  axis. 
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Figure  40.  Reaper:  Technical  Maturity 
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Figure  41.  Reaper:  Drawings  Completed 

Note.  Data  for  2010  are  missing  from  the  GAO  report. 
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Figure  42.  Reaper:  Number  of  Sentences  That  Mention  Software 

in  the  GAO  Reports 

G.  Global  Hawk:  2004-2011  (Eight  Years) 

The  yearly  schedules  and  milestones  for  the  Global  Hawk  program  are  shown 
in  Error!  Reference  source  not  found..  Every  published  schedule  is  shown  as  a 
row  in  the  table.  The  columns  represent  the  different  years  from  the  schedules 
shown  in  the  GAO  reports.  For  example,  in  2004,  IOC  was  scheduled  for  December 
2005  and  the  full-rate  production  decision  was  planned  for  November  2006. 

This  chart  is  useful  because  it  clearly  shows  schedule  slip  as  a  function  of 
time.  There  was  a  Nunn-McCurdy  breach  in  2005,  but  it  wasn’t  reflected  into  the 
write-up  until  2007. 

Error!  Reference  source  not  found,  shows  the  time  to  a  major  milestone 
measured  in  months  from  development  start.  The  reported  “acquisition  cycle  time” 
was  measured  from  development  start  to  IOC,  which  was  planned  to  occur  before 
the  FRP  decision.  The  acquisition  cycle  time  remained  57  months  until  2007  when  it 
increased  to  78  months.  Thereafter,  it  was  reported  as  TBD  (to  be  defined). 

The  system  was  already  in  LRIP  production  (although  one  could  argue  that 
the  planes  were  really  EDMs).  There  was  a  very  large  increase  in  the  schedule  in 
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2007  when  the  time  to  FRP  jumped  from  74  months  in  2006  to  97  months  in  2007  (a 
31%  increase).  There  was  a  small  increase  in  the  schedule  in  2009. 

Error!  Reference  source  not  found,  shows  the  R&D  costs  for  the  program. 
All  costs  have  been  adjusted  for  inflation.  In  the  figure,  the  planned  total  program 
cost  is  shown  as  the  black  line  with  squares. 

The  R&D  funding  needed  to  complete  the  program,  listed  in  the  “Program 
Essentials”  block  in  each  GAO  report,  is  shown  as  the  green  line  with  triangles.  The 
GAO  reports  did  not  report  a  yearly  R&D  spending,  so  it  was  derived  from  the 
reported  cost  data.  The  red  line  with  asterisks  is  the  actual  R&D  spending  for  each 
year.  The  blue  line  with  diamonds  is  the  actual  R&D  spending  from  the  beginning  of 
the  program. 

Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs,20  and  the  program  unit  costs.21  The  planned  quantity  was 
flat  at  51  units  from  2004-2006  and  was  then  flat  at  54  units  from  2007-2010.  The 
quantity  increased  43%  in  201 1  to  77  units.  While  the  program  unit  cost  decreased 
5%  in  201 1  due  to  the  increase  in  quantity,  the  procurement  unit  cost  actually  rose 
7%.  Normally,  procurement  unit  costs  should  decrease  due  to  learning  curve  effects 
and  fixed  overhead  expenses  being  spread  out  over  more  units. 

Error!  Reference  source  not  found,  shows  the  technical  maturity  for  each 
year  based  on  the  GAO  reports.  Initially,  14  technologies  were  reported  as  critical, 
but  the  number  dropped  to  10  in  2007.  All  were  identified  as  mature  starting  in  2007. 

Error!  Reference  source  not  found,  shows  the  number  of  design  drawings 
completed  for  each  year.  Data  for  2006  were  unclear  and  are  not  plotted.  The  2005 
GAO  report  states  that  90%  of  the  engineering  drawings  were  completed  by  late  FY 

20  The  procurement  unit  cost  is  the  total  procurement  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  Procurement  unit  cost  does  not 
include  R&D  costs. 

21  The  program  unit  cost  is  the  total  program  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  The  total  program  costs  are  the 
R&D  costs  plus  the  procurement  costs. 
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2004.  The  reports  stated  100%  in  2007  and  thereafter,  although  the  2008  report 
stated,  “however,  frequent  and  substantive  engineering  changes  increased 
development  and  airframe  costs  and  delayed  delivery  and  testing  schedules.”  So 
while  the  drawings  were  “released,”  they  were  far  from  being  mature. 

The  write-ups  do  not  have  a  specific  measure  of  software  maturity.  As  an 
approximation,  the  number  of  sentences  in  each  report  mentioning  software  was 
counted  as  a  gauge  of  the  software  development.  Error!  Reference  source  not 
found,  shows  the  number  of  lines  in  the  GAO  reports  that  mention  software. 
Software  was  mentioned  twice  per  year  for  the  years  2007-2010. 
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Figure  43.  Global  Hawk  Schedules 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  DevStart  (development 
start)  are  shown.  The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells 
are  planned  to  occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred 
at  the  time  of  the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at 
the  time  of  the  report  for  the  years  after  the  GAO  report. 
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Figure  44.  Global  Hawk:  Time  to  FRP 

Note.  The  red  line  with  triangles  is  the  program  acquisition  cycle  time  listed  in  the  GAO  write-ups  (in 
months).  It  was  measured  from  development  start  to  IOC,  which  was  planned  to  occur  before  the 
FRP  decision.  After  2007,  it  was  listed  as  “TBD.”  The  black  line  with  boxes  is  the  time  from 
development  start  to  FRP  (in  months). 
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Figure  45.  Global  Hawk:  R&D  Costs  By  Year  (Derived  From  the  GAO  Data) 

Note.  The  green  line  with  triangles  is  the  R&D  funding  needed  to  complete  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  black  line  with  boxes  is  the  planned  total 
R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  The  red  line  with  black 
asterisks  is  the  R&D  spending  during  the  year.  The  blue  line  with  diamonds  is  the  Total  R&D 
spending  from  the  beginning  of  the  project.  It  is  derived  from  the  “funding  needed  to  complete”  and 
the  “R&D  cost”  categories  in  the  reports. 
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Figure  46.  Global  Hawk:  Production  Quantity  and  Unit  Cost 

Note.  The  black  line  with  squares  is  the  procurement  unit  cost  adjusted  for  inflation.  The  red  line  with 
triangles  is  the  program  unit  cost  adjusted  for  inflation  (the  program  unit  cost  includes  the  R&D  costs 
amortized  over  the  total  production  lot). 


Figure  47.  Global  Hawk:  Technical  Maturity 
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Figure  48.  Global  Hawk:  Drawings  Completed 

Note.  The  number  of  drawings  completed  was  not  reported  in  2006. 


Figure  49.  Global  Hawk:  Number  of  Lines  Mentioning  Software 


H.  Space  Based  Infrared  System  (SBIRS)  High:  2003-2011 
(Nine  Years) 

A  historical  look  at  the  major  SBIRS  milestones  by  year  is  shown  in  Error! 
Reference  source  not  found..  Every  published  schedule  is  shown  as  a  row  in  the 
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table.  The  columns  represent  the  different  years  in  each  program  schedule  from  the 
GAO  reports.  In  2003,  the  first  satellite  launch  was  referred  to  as  LRIP.  For  the 
following  years,  IOC  was  used  to  denote  the  first  satellite  delivery.  For  example,  in 
2003,  satellite  launch  (LRIP)  was  scheduled  for  October  2006.  In  2004,  IOC  was 
planned  for  September  2006.  There  was  a  Nunn-McCurdy  breach  in  May  2004, 
which  shows  up  as  a  large  program  unit  cost  increase  in  2006  (the  increase  from 
2004  to  2006  was  81%). 

Error!  Reference  source  not  found,  shows  the  R&D  costs  for  the  program. 
All  costs  have  been  adjusted  for  inflation.  The  GAO  reports  did  not  report  a  yearly 
R&D  spending  amount,  so  it  was  derived  from  the  reported  cost  data.  In  the  figure, 
the  planned  total  program  cost  is  shown  as  the  black  line  with  squares. 

The  R&D  funding  needed  to  complete  the  program,  listed  in  the  “Program 
Essentials”  block  in  each  GAO  report,  is  shown  as  the  green  line  with  triangles. 

The  red  line  with  asterisks  is  the  actual  R&D  spending  for  each  year.  The  blue 
line  with  diamonds  is  the  actual  R&D  spending  from  the  beginning  of  the  program. 

Error!  Reference  source  not  found,  shows  the  total  production  quantity,  the 
procurement  unit  costs,22  and  the  program  unit  costs.23  Because  the  system  is  a 
specialized  satellite,  the  production  quantity  is  very  low  and,  thus,  there  can  very 
large  swings  in  the  unit  cost. 

Figure  53  shows  the  technical  maturity  for  each  year  based  on  the  GAO 
reports.  Three  critical  technologies  were  identified  for  the  program,  and  it  was 
reported  that  all  technologies  were  mature  for  2003-2008  and  2010.  In  the  2009 
report,  the  GAO  report  stated  that  the  program  had  split  one  technology  into  two, 
one  of  which  was  immature. 

22  The  procurement  unit  cost  is  the  total  procurement  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  Procurement  unit  cost  does  not 
include  R&D  costs. 

23  The  program  unit  cost  is  the  total  program  cost  divided  by  the  total  quantity,  which  are  both 
identified  in  the  “Program  Performance”  block  of  each  GAO  write-up.  The  total  program  costs  are  the 
R&D  costs  plus  the  procurement  costs. 
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Error!  Reference  source  not  found,  shows  the  number  of  completed 
drawings.  The  data  are  not  consistent  with  the  schedule  slip  and  the  continued 
increase  in  spending  from  2003-201 1 .  RDT&E  spending  was  $1 .52  billion,  even 
though  there  wasn’t  a  change  in  the  “number  of  drawings  completed.”  The  relatively 
flat  percentage  of  drawings  completed  (from  2009-201 1 )  is  also  not  consistent  with 
the  continued  increase  in  RDT&E  costs  during  those  years. 

The  write-ups  do  not  have  a  specific  measure  of  software  maturity.  As  an 
approximation,  the  number  of  sentences  in  each  report  mentioning  software  was 
counted  as  a  gauge  of  the  software  development.  Error!  Reference  source  not 
found,  shows  the  number  of  lines  in  the  GAO  reports  that  mention  software.  The 
GAO  write-ups  mention  software  at  least  five  times  from  2008-201 1 ,  which  is  much 
more  often  than  the  typical  GAO  program  write-up. 
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Figure  50.  SBIRS  Schedule 

Note.  Each  yearly  schedule  from  the  GAO  reports  is  shown  as  a  row  in  the  table.  Milestones  such  as 
DR  (design  review),  LRIP,  FRP,  IOC,  Nunn  (Nunn-McCurdy  breach),  and  DevStart  (development 
start)  are  shown.  The  white  cells  are  the  year  of  the  report  and  milestones  shown  in  the  white  cells 
are  planned  to  occur  during  the  year.  The  milestones  shown  in  the  green  cells  had  already  occurred 
at  the  time  of  the  GAO  report.  The  milestones  shown  in  the  yellow  cells  were  planned  milestones  at 
the  time  of  the  report  for  the  years  after  the  GAO  report. 
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Figure  51.  SBIRS:  Cost  Profile 

Note.  The  green  line  with  triangles  is  the  R&D  funding  needed  to  complete  the  program  and  is  from 
the  “Program  Essentials”  section  of  each  GAO  report.  The  black  line  with  boxes  is  the  planned  total 
R&D  cost  taken  from  the  “Program  Performance”  section  of  each  report.  The  red  line  with  black 
asterisks  is  the  R&D  spending  during  the  year.  The  blue  line  with  diamonds  is  the  Total  R&D 
spending  from  the  beginning  of  the  project.  It  is  derived  from  the  “funding  needed  to  complete”  and 
the  “R&D  cost”  categories  in  the  reports. 


Figure  52.  SBIRS:  Quantity  and  Unit  Costs 

Note.  The  black  line  with  squares  is  the  procurement  unit  cost  adjusted  for  inflation.  The  red  line  with 
triangles  is  the  program  unit  cost  adjusted  for  inflation  (the  program  unit  cost  includes  the  R&D  costs 
amortized  over  the  total  production  lot). 
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Figure  53.  SBIRS:  Technical  Maturity 


Figure  54.  SBIRS  Drawings  Completed 
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Figure  55.  SBIRS:  Lines  in  the  GAO  Reports  That  Mentioned  Software 
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V. 


Summary  and  Recommendations 


Risk  management  is  one  of  the  eight  technical  management  processes  in  the 
overall  systems  engineering  process.  The  process  of  risk  management  provides  a 
framework  for  risks  to  be  identified,  analyzed,  mitigated,  and  monitored.  For 
example,  defining  stakeholder  needs  early  in  development  reduces  the  chances  of 
building  the  wrong  system.  Requirements  analysis  reduces  the  risk  of  a  constantly 
changing  functional  baseline.  Testing  LRIP  units  as  part  of  IOT&E  reduces  the  risk 
of  costly  upgrades  to  full-rate  production  units.  The  list  of  mitigation  activities  is 
unbounded.  It  is  not  possible  to  examine  every  activity  and  its  impact  on  risk 
reduction. 

This  research  paper  addresses  risk  mitigation  activities  that  are  considered 
best  practices  and  that  are  documented  in  the  yearly  GAO  “Assessments  of 
Selected  Weapon  Programs”  reports.  The  GAO  has  spent  a  considerable  amount  of 
time  every  year  evaluating  several  dozen  major  weapon  system  programs.  This 
research  project  is  based  on  the  prior  work  by  the  GAO  office.  Namely,  this  project 
has  involved  the  analysis  and  evaluation  of  eight  major  weapon  system  programs 
over  a  nine-year  period  (2003-201 1).  Changes  in  cost,  schedule,  and  performance 
have  been  tracked  over  time,  which  provides  a  unique  perspective  to  understanding 
program  execution. 

This  project  has  focused  on  some  key  systems  engineering  principles  that  are 
risk-mitigation  activities.  The  best  practices  that  have  been  evaluated  in  this 
research  paper  are  the  following: 

■  Use  of  technical  readiness  levels24  to  prevent  immature  technologies 
from  being  incorporated  into  new  system  designs  since  there  is  a  high 
risk  of  cost  overruns  and  schedule  delays.  This  metric  measures  the 
maturity  of  technology  being  considered  for  use  in  a  new  weapon 
system. 


24  The  GAO  reports  did  not  mention  TRLs  specifically,  but  they  did  identify  the  maturity  of  the  critical 
technologies. 
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■  “Design  drawings  completed”  is  an  important  entrance  criterion  to  the 
Critical  Design  Review.  This  metric  measures  the  maturity  of  a 
system’s  design  before  prototypes  are  built  and  tested. 

■  Earned  Value  Management  allows  the  government  to  monitor  cost  and 
schedule  performance  by  the  prime  contractor.  EVM  provides  an  early 
and  in-depth  evaluation  of  progress  by  the  prime  contractor  in  the 
design  of  a  new  weapon  system. 

■  Production  maturity  evaluates  the  effectiveness  of  production  activities 
and  the  potential  for  cost  increases  and/or  schedule  delays. 

One  important  best  practice  not  directly  addressed  in  the  GAO  reports  is 
software  development  and  test. 

A.  Technical  Maturity 

Technology  Readiness  Levels  (TRLs)  are  a  measure  used  by  the  United 
States  Department  of  Defense,  other  government  agencies,  and  many  of  the  world's 
major  companies  to  assess  the  maturity  of  evolving  technologies  (materials, 
components,  devices,  etc.)  prior  to  incorporating  that  technology  into  a  new  system. 
Generally  speaking,  when  a  new  technology  is  first  invented  or  conceptualized,  it  is 
not  suitable  for  immediate  application.  Instead,  new  technologies  are  usually 
subjected  to  experimentation,  refinement,  and  increasingly  realistic  testing.  Once  the 
technology  is  sufficiently  proven,  it  can  be  incorporated  into  a  system/subsystem 
(“Technology  Readiness  Level,”  2012). 

Unfortunately,  based  on  this  research  project,  TRLs  are  not  a  good  indicator 
of  program  performance.  Table  1  shows  the  number  of  technologies  identified  as 
mature  versus  the  total  number  identified  based  on  the  yearly  GAO  reports.  For 
example,  “4/14”  means  that  four  of  14  technologies  were  identified  as  mature  in  the 
program  write-ups.  Except  for  JSF  and  JTRS,  all  of  the  other  programs  identified  the 
technologies  as  mature,  even  though  there  were  significant  delays  and  cost 
overruns  that  occurred  after  the  technologies  were  identified  as  mature.  The  cost 
increases  in  the  program  unit  costs  and  the  procurement  unit  costs  are  shown  in 
Error!  Reference  source  not  found.. 
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For  example,  the  EFV  program  reported  that  all  critical  technologies  were 
mature  in  2005.  From  2005-2011,  the  program  unit  cost  increased  by  122%  and  the 
procurement  unit  cost  increased  by  106%.  The  Excalibur,  C-5  RERP,  and  SBIRS 
programs  saw  large  increases  in  unit  costs.  The  Reaper  and  Global  Hawk  effectively 
had  no  increase  in  program  unit  costs  and  showed  17%  and  13%  increases  in 
procurement  unit  costs,  respectively.  The  Reaper  had  a  large  increase  in  units, 
which  helped  to  keep  the  unit  cost  increases  lower. 

Table  1.  Technical  Maturity  Levels  for  Eight  Programs 


*  7 
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Table  2.  Program  and  Procurement  Unit  Cost  Increases 


Program 

Period 

Program  Unit 

Cost  Increase 

Procurement 

Unit  Cost 

Increase 

Excalibur 

2006  to  2011 

224% 

136% 

EFV 

2005  to  2011 

122% 

106% 

C-5  RERP 

2004  to  201 1 

52% 

38% 

JTRS  Cluster  1 

Did  not  reach  full  ma 

turity 

JSF 

Did  not  reach  full  maturity 

Reaper 

2008  to  2011 

0% 

17% 

Global  Hawk 

2007  to  201 1 

-2% 

13% 

SBIRS 

2004  to  2011 

35% 

167% 

B.  Design  Stability 

Use  of  “design  drawings  completed”  is  an  important  entrance  criterion  to  the 
Critical  Design  Review.  This  metric  measures  the  maturity  of  system’s  design  before 
prototypes  are  built  and  tested.  According  to  the  DAG  (Defense  Acquisition 
University  [DAU],  2012), 

The  CDR  should  be  conducted  when  the  product  baseline  has  been 
achieved,  allowing  fabrication  of  hardware  and  coding  of  software 
deliverables  to  proceed.  A  rule  of  thumb  is  that  75  percent  to  90  percent  of 
(manufacturing  quality)  product  drawings  ...  are  complete,  (p.  267) 

Observations  from  the  eight  programs  studied  are  as  follows: 

1 .  Some  of  the  values  are  questionable  because  they  are  an  exact 
multiple  of  10%. 

2.  In  many  cases,  the  “percent  of  drawings  completed”  decreased  after 
obtaining  98%  or  higher  in  a  previous  year.  This  suggests  that  a  lot  of 
redesign  had  taken  place,  that  the  measurement  of  the  original  value 
was  poor,  and/or  that  the  original  release  of  a  drawing  was  before  the 
design  had  stabilized. 

3.  The  number  of  drawings  released  may  be  an  early  indicator  of  large 
cost  over-runs  and  schedule  delays.  For  example,  for  the  C-5  RERP 
program,  the  percent  of  drawings  released  dropped  in  2007,  but  it 
wasn’t  until  2008  when  the  time  to  FRP  changed. 
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Regarding  the  phrase  “product  drawings  are  complete,”  there  are  many 
different  valid  definitions.  Ideally,  documents  “complete”  should  mean 

1.  the  design  and  all  supporting  information  have  been  completed, 

2.  the  drawing  has  been  developed, 

3.  the  drawing  has  been  reviewed  and  approved  by  the  appropriate 
people, 

4.  the  drawing  has  been  made  part  of  the  physical  baseline, 

5.  the  drawing  has  been  released  to  manufacturing, 

6.  a  supplier  and  cost  data  have  been  identified,  and 

7.  the  prototype  has  or  will  be  manufactured  using  the  “completed 
drawing.” 

Another  wrinkle  into  the  use  of  “drawings  complete”  is  when  an  engineering 
change  proposal  (ECP)  occurs.  How  changes  to  existing  drawings  and  the  addition 
of  new  drawings  are  reported  will  impact  the  reported  percentage  of  drawings 
completed. 

Metrics  that  would  aid  in  understanding  the  design  maturity  of  a  program 
include  the  following: 

■  drawings  released, 

■  drawings  modified  and  re-released  (gross  number  and  percentage  of 
existing  drawings), 

■  new  drawings  (number  and  percentage  of  existing  drawings),  and 

■  number  of  obsolete  drawings. 

It  is  also  important  to  track  the  severity  of  any  changes  in  a  drawing.  For 
example,  fixing  a  typo  in  the  notes  section  of  a  drawing  is  not  as  significant  as 
changing  the  design  of  the  part,  material  callouts,  tolerances,  and  so  forth.  The  best 
measure  of  this  will  be  if  there  is  a  change  in  “fit,  form  or  function.”  A  way  to  easily 
measure  that  is  the  number  of  new  part  numbers. 
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An  important  additional  metric  that  is  not  directly  related  to  drawings  is  the 
number  of  modified  or  new  requirements.  New  requirements  could  cause  large  re¬ 
design  efforts,  which  could  affect  the  number  of  released  drawings. 

C.  Earned  Value 

Earned  value  management  (EVM)  is  a  project  management  technique  for 
measuring  project  performance  in  an  objective  manner.  EVM  is  able  to  provide 
accurate  forecasts  of  project  performance  problems,  which  enables  proactive  project 
management. 

Earned  value  will  be  required25  on  all  of  the  programs  in  the  reports  because 
of  the  cost  of  the  development  or  production.  The  requirement  for  EVM  applies  to 
cost  or  incentive  contracts,  subcontracts,  intra-government  work  agreements,  and 
other  agreements  that  meet  the  dollar  thresholds  prescribed  in  DoD  Instruction 
5000.02  (DoD,  2008,  p.44). 

There  were  only  a  few  programs  that  mentioned  EVM  in  the  GAO  reports. 

2003  ->  no  programs 

2004  ->  no  programs 

2005  ->  one  program 

2006  ->  two  programs 

2007  ->  four  programs 

2008  ->  six  programs 

2009  ->  six  programs 

25  The  application  thresholds  (total  contract  value  including  planned  options  in  then-year  dollars)  are 
as  follows:  $20  million  but  less  than  $50  million — EVM  implementation  compliant  with  the  guidelines 
in  ANSI/EIA-748  is  required.  No  formal  Earned  Value  Management  System  (EVMS)  validation  is 
required.  If  $50  million  or  greater,  then  EVM  implementation  compliant  with  the  guidelines  in 
ANSI/EIA-748  is  required.  An  EVMS  that  has  been  formally  validated  and  accepted  by  the 
responsible  contracting  officer  is  required. 
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2010  ->  one  program 

201 1  two  programs 

Even  when  mentioned,  the  data  were  often  vague:  for  example,  from  the 
2004  GAO  report  on  major  weapon  systems  (GAO-04-248,  p.  66),  “Contract 
performance  data  indicates  that  work  is  slightly  behind  schedule  and  over  cost.” 

In  some  cases,  the  EVM  system  was  broken:  for  example,  in  the  2007  GAO 
report  on  major  weapon  systems  (GAO-07-406SP): 

In  March  2006,  the  lead  contractor  lost  its  earned  value  management 
certification  due  to  a  recent  compliance  review  that  found  lack  of  progress  in 
addressing  long-standing  systemic  deficiencies.  Without  certified  earned 
value  management  data,  the  Army  will  not  have  timely  information  on  the 
contractor’s  ability  to  perform  work  within  estimated  cost  and  schedule. 
According  to  the  program  office,  the  contractor  did  not  make  its  first  milestone 
detailed  in  the  Defense  Contract  Management  Agency’s  corrective  action 
plans  in  efforts  to  obtain  earned  value  compliance.  Still,  the  contractor  plans 
to  be  compliant  by  the  end  of  August  2007,  3  months  after  ARH  low-rate  initial 
production  is  scheduled  to  begin,  (p.  38) 

D.  Production  Maturity 

Statistical  Process  Control  (SPC)  tools  are  a  best  practice,  but  they  work  best 
in  production,  usually  on  quantities  in  the  many  hundreds  or  more.  Prior  to  actual 
production,  a  Production  Readiness  Review  (PRR)  will  be  an  early  indicator  of 
potential  manufacturing  problems.  Manufacturing  readiness  levels  are  useful  for 
quantifying  the  maturity  level  of  the  manufacturing  processes. 

There  are  many  questions  that  are  important  for  manufacturing  a  system: 

■  Can  the  system  be  built  per  cost  target? 

■  Can  it  be  built  to  the  schedule? 

■  Is  the  supplier  base  in  place  and  stable? 

■  What’s  the  lead  time? 

■  Does  the  contractor  have  a  trained  workforce? 

■  Does  the  contractor  have  the  necessary  facilities? 

■  What  is  the  contractor’s  quality  management  system? 
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Does  the  prime  contractor  have  a  management  plan  for  managing 
subcontractors? 


There  is  a  good  write-up  on  this  topic  in  the  Defense  Acquisition  Guidebook  in 
Section  11.3.3.1  (DAU,  2012,  p.  935): 

The  quality  management  process  begins  early  in  the  life  cycle  and  continues 
throughout.  The  principal  elements  of  the  quality  management  process 
include: 

■  Objectively  evaluating  performed  processes,  work  products, 
product/process  design  and  services  against  the  applicable  process 
descriptions,  standards,  procedures,  policies,  and  documented 
expectations; 

■  Understanding  the  full  scope  of  customer  requirements,  assessing 
risks  associated  with  meeting  those  requirements,  and  verifying  that 
they  are  satisfied; 

■  Identifying  and  documenting  noncompliance  issues,  especially  those 
affecting  cost,  schedule,  productivity,  and  performance; 

■  Using  tools  and  techniques  in  a  disciplined  manner  to  determine  root 
causes  of  noncompliance  issues; 

■  Addressing  noncompliance  issues  by  initiating  and  tracking  corrective 
and  preventative  actions  to  assure  the  root  cause(s)  of  the 
defect/deficiency  has  been  identified  and  removed;  and 

■  Providing  feedback  to  program  managers,  their  staff,  and  corporate 
managers  to  identify  lessons  learned,  improve  process  robustness  for 
future  projects,  and  evaluate  trends. 

So  even  if  a  contractor  is  not  using  SPC,  they  need  to  have  some  data  about 
how  well  they’re  doing  in  the  above  activities.  Something  as  simple  as  %scrap  and 
%rework  will  give  a  lot  of  insight  into  the  production  effectiveness.  If  contractors  can’t 
quantify  those,  then  it’s  impossible  for  them  to  accurately  predict  the  cost  of  future 
production. 

E.  Software 

There  wasn’t  enough  attention  given  to  software  in  the  GAO  reports.  Design 
maturity  is  not  just  the  number  of  drawings  released.  That  is  only  a  measure  of  the 
physical  hardware  design.  Software  is  usually  very  important  and  risky  on  large 
weapon  programs.  Here  are  some  statistics  from  the  DAU  level  III  course  on 
systems  engineering  [SYS  302]  (DAU,  2011;  a  little  dated  but  still  relevant): 
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■  64%  of  DoD  software  projects  did  not  meet  time/budget  goals; 

■  Half  of  projects  overran  cost  estimates  by  43%;  and 

■  15%  of  projects  were  canceled. 

Additional  information  on  software  from  the  DAU’s  SYS-302  class  is  as  follows: 

■  Software  controls  most  of  today’s  systems  key  functionalities; 

■  Software  is  usually  on  the  critical  path;  and 

■  Software  is  a  complex,  conformable,  changeable,  and  invisible 
product. 

On  average,26  there  were  2.1  sentences  devoted  to  software  in  the  GAO 
reports  studied  (i.e. ,  eight  programs  over  nine  years).  With  the  possible  exception  of 
the  Excalibur  program,  software  is  a  major  component  of  the  programs  studied. 

For  example,  the  importance  of  software  is  clearly  stated  in  the  SBIRS  write¬ 
up  for  2008.  Prior  to  2008,  there  were  only  a  total  of  three  sentences  devoted  to 
software.  However,  there  were  seven  sentences  in  2008  and  six  sentences  in  2009. 
(Write-ups  were  usually  around  38  sentences  long,  not  including  the  cost  data). 

One  part  of  the  SBIRS  write-up  in  the  2009  GAO  report  on  major  weapon 
systems  (GAO-09-326SP)  summed  it  up  as  follows: 

Design  is  considered  stable  since  about  97  percent  of  expected  design 
drawings  are  releasable.  However,  the  program  has  experienced  design- 
related  problems  and  more  could  emerge.  For  example,  the  flight  software 
that  controls  the  health  and  status  of  the  space  vehicle  was  found  to  be 
inadequate  when  it  unexpectedly  failed  during  testing  in  2007.  In  April  2008, 
independent  experts  approved  a  new  software  design.  DOD  estimates  the 
design  changes  will  delay  the  first  satellite  launch  at  least  15  months  to 
December  2009  and  increase  costs  by  about  $414  million.  Further  cost 
increases  and  schedule  delays  are  likely,  (p.  136) 

This  says  that  the  problems  with  software  will  cost  $414  million  or  more. 

Clearly,  the  problems  with  software  were  hidden  during  the  period  2003-2007 

because  only  three  sentences  mentioned  software  in  five  years  of  reports. 


26  Excalibur  was  excluded  from  the  average. 
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It  should  not  be  difficult  to  acquire  software  metrics.  Any  company  that  claims 
to  be  CMMI27  (CMMI  Institute,  2013)  Level  2  or  higher  will  be  using  metrics  to 
manage  their  software  projects.  So  theoretically,  performance  metrics  should  exist 
and  should  be  reported.  Some  key  software  metrics  include  the  following: 

■  requirement  count, 

■  number  of  changes, 

■  coding  size,28 

■  units  tested,  and 

■  defect  count. 

If  the  contractor  is  CMMI  2  or  higher,  then  they  should  have  identified  metrics 
in  their  proposal  and  would  be  contractually  obligated  to  generate  and  use  the 
metrics. 

F.  Final  Comments 

There  are  various  situations  reported  that  deserve  closer  scrutiny,  such  as  the 
following: 

1 .  increases  in  procurement  unit  costs  when  quantities  increase, 

2.  a  drop  in  the  percent  of  drawings  completed, 

3.  “drawings  completed”  that  stay  the  same  even  when  R&D  spending  is 
high, 

4.  “drawings  completed”  that  conveniently  fall  on  a  multiple  of  10%  (e.g., 
80%), 

5.  lack  of  information  on  software  development,  and 

6.  lack  of  information  on  T&E  activities  and  accomplishments. 


27  Capability  Maturity  Model  Integration  (CMMI)  is  a  process  improvement  approach  whose  goal  is  to 
help  organizations  improve  their  performance. 

28 

Lines  of  code  is  an  easy  way  to  measure  the  size  of  the  software  development,  but  it  is  not  very 
relevant.  Function  points  or  configuration  items  are  better  measures. 
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Appendix  A.  Technology  Readiness  Levels 


Table  A1.  Technology  Readiness  Levels  in  the  Department  of  Defense  (DoD) 

(“Technology  Readiness  Level,”  2012) 


Technology  Readiness  Levels  in  the  Department  of  Defense  (DoD) 

Technology 
Readiness  Level 

Description 

Supporting  Information 

1.  Basic  principles 
observed  and  reported 

Lowest  level  of  technology  readiness. 
Scientific  research  begins  to  be 
translated  into  applied  research  and 
development  (R&D).  Examples  might 
include  paper  studies  of  a  technology’s 
basic  properties. 

Published  research  that  identifies  the 
principles  that  underlie  this  technology. 
References  to  who,  where,  when. 

2.  Technology  concept 
and/or  application 
formulated 

Invention  begins.  Once  basic  principles 
are  observed,  practical  applications 
can  be  invented.  Applications  are 
speculative,  and  there  may  be  no  proof 
or  detailed  analysis  to  support  the 
assumptions.  Examples  are  limited  to 
analytic  studies. 

Publications  or  other  references  that  out¬ 
line  the  application  being  considered  and 
that  provide  analysis  to  support  the 
concept. 

3.  Analytical  and 
experimental  critical 
function  and/or 
characteristic  proof  of 
concept 

Active  R&D  is  initiated.  This  includes 
analytical  studies  and  laboratory 
studies  to  physically  validate  the 
analytical  predictions  of  separate 
elements  of  the  technology.  Examples 
include  components  that  are  not  yet 
integrated  or  representative. 

Results  of  laboratory  tests  performed  to 
measure  parameters  of  interest  and 
comparison  to  analytical  predictions  for 
critical  subsystems.  References  to  who, 
where,  and  when  these  tests  and 
comparisons  were  performed. 

4.  Component  and/or 
breadboard  validation  in 
laboratory  environment 

Basic  technological  components  are 
integrated  to  establish  that  they  will 
work  together.  This  is  relatively  “low 
fidelity”  compared  with  the  eventual 
system.  Examples  include  integration 
of  “ad  hoc”  hardware  in  the  laboratory. 

System  concepts  that  have  been 
considered  and  results  from  testing 
laboratory-scale  breadboard(s). 

References  to  who  did  this  work  and 
when.  Provide  an  estimate  of  how 
breadboard  hardware  and  test  results 
differ  from  the  expected  system  goals. 

5.  Component  and/or 
breadboard  validation  in 
relevant  environment 

Fidelity  of  breadboard  technology 
increases  significantly.  The  basic 
technological  components  are 
integrated  with  reasonably  realistic 
supporting  elements  so  they  can  be 
tested  in  a  simulated  environment. 
Examples  include  “high-fidelity” 
laboratory  integration  of  components. 

Results  from  testing  laboratory 
breadboard  system  are  integrated  with 
other  supporting  elements  in  a  simulated 
operational  environment.  How  does  the 
“relevant  environment”  differ  from  the 
expected  operational  environment?  How 
do  the  test  results  compare  with 
expectations?  What  problems,  if  any, 
were  encountered?  Was  the  breadboard 
system  refined  to  more  nearly  match  the 
expected  system  goals? 

6.  System/subsystem 

Representative  model  or  prototype 

Results  from  laboratory  testing  of  a 
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model  or  prototype 
demonstration  in  a 
relevant  environment 

system,  which  is  well  beyond  that  of 

TRL  5,  is  tested  in  a  relevant 
environment.  Represents  a  major  step 
up  in  a  technology’s  demonstrated 
readiness.  Examples  include  testing  a 
prototype  in  a  high-fidelity  laboratory 
environment  or  in  a  simulated 
operational  environment. 

prototype  system  that  is  near  the  desired 
con-figuration  in  terms  of  performance, 
weight,  and  volume.  How  did  the  test 
environment  differ  from  the  operational 
environment?  Who  performed  the  tests? 
How  did  the  test  compare  with 
expectations?  What  problems,  if  any, 
were  encountered?  What  are/were  the 
plans,  options,  or  actions  to  resolve 
problems  before  moving  to  the  next  level? 

7.  System  prototype 
demonstration  in  an 
operational 
environment. 

Prototype  near  or  at  planned 
operational  system.  Represents  a 
major  step  up  from  TRL  6  by  requiring 
demonstration  of  an  actual  system 
prototype  in  an  operational 
environment  (e.g.,  in  an  air-craft,  in  a 
vehicle,  or  in  space). 

Results  from  testing  a  prototype  system  in 
an  operational  environment.  Who 
performed  the  tests?  How  did  the  test 
compare  with  expectations?  What 
problems,  if  any,  were  encountered? 

What  are/were  the  plans,  options,  or 
actions  to  resolve  problems  before 
moving  to  the  next  level? 

8.  Actual  system 
completed  and  qualified 
through  test  and 
demonstration. 

Technology  has  been  proven  to  work 
in  its  final  form  and  under  expected 
conditions.  In  almost  all  cases,  this 

TRL  represents  the  end  of  true  system 
development.  Examples  include 
developmental  test  and  evaluation 
(DT&E)  of  the  system  in  its  intended 
weapon  system  to  determine  if  it  meets 
design  specifications. 

Results  of  testing  the  system  in  its  final 
configuration  under  the  expected  range  of 
environmental  conditions  in  which  it  will 
be  expected  to  operate.  Assessment  of 
whether  it  will  meet  its  operational 
requirements.  What  problems,  if  any, 
were  encountered?  What  are/were  the 
plans,  options,  or  actions  to  resolve 
problems  before  finalizing  the  design? 

9.  Actual  system  proven 
through  successful 
mission  operations. 

Actual  application  of  the  technology  in 
its  final  form  and  under  mission 
conditions,  such  as  those  encountered 
in  operational  test  and  evaluation 
(OT&E).  Examples  include  using  the 
system  under  operational  mission 
conditions. 

OT&E  reports. 
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Appendix  B.  Example  of  a  GAO  Report  on  a  Major 

Weapon  System  (GAO,  2011,  p.  69) 


Common  Name:  Global  Hawk 


Global  Hawk  (RQ-4A/B) 

The  Air  Force's  Global  Hawk  is  a  high-altitude,  long- 
endurance  unmanned  aircraft  with  integrated 
sensors  and  ground  stations  providing  intelligence, 
surveillance,  and  reconnaissance  capabilities.  The 
Global  Hawk  will  replace  the  U-2.  After  a  successful 
technology  demonstration,  the  system  entered 
development  and  limited  production  in  March  2001. 

The  progrant  includes  RQ-4A  aircraft  similar  to  the 
original  demonstrators,  as  well  as  larger  and  more 
capable  RQ-4Bs.  We  assessed  the  RQ-1B.  which  is 
being  procured  in  three  blocks. 


Source  No  rtf-cop  Qrifwmr 


Concept 

System  development  Production 

A 

▲ 

A 

A 

Demonstration 

Development  start  / 

GAO 

Last 

program  start 

low-rate  decision 

review 

procurement 

<2/94) 

<3/01) 

(11/10) 

(2018) 

Program  Essentials 

Program  Performance  (fiscal  year  2011  dollars  In  millions) 

Prime  contraclor:  Northrop  Grumman 

As  of 

Latest 

Percent 

Program  office:  Wright-Pattereon  AFB. 

03/2001 

07/2010 

change 

OH 

Besearch  and  development  cost 

SI  .026.3 

S3. 948 .7 

284.8 

Funding  needed  to  complete: 

Procurement  cost 

S4 .255.0 

S9.481.6 

122.8 

R&D:  $1,067.5  million 

Total  program  cost 

S5 .312.4 

$13,575.7 

155.5 

Procurement:  $5,339.1  million 

Program  unit  cost 

$84,323 

$176,307 

109  1 

Total  funding:  $6,406.6  million 

Total  quantities 

63 

77 

222 

Procurement  quantity:  39 

Acquisition  cycle  lime  (months) 

55 

TBD 

NA 

The  Global  Hawk  RQ-4B  has  mature  critical 
technologies,  a  stable  design,  and  proven 
production  processes,  but  it  remains  at  risk  for 
late  design  changes  and  costly  retrofits.  The 
completion  of  operational  tests  for  the  aircraft 
that  make  up  the  largest  part  of  the  program  has 
been  delayed  nearly  4  years  by  testing  discoveries, 
concurrent  testing,  resource  constraints,  and 
weather  problems.  The  program  will  have 
procured  more  than  half  of  those  aircraft  by  the 
time  testing  is  complete  in  December  2010.  The 
program  also  plans  to  procure  more  than  half  the 
aircraft  with  advanced  radar  before  it  completes 
operational  testing  in  2013.  The  Air  Force  is  taking 
steps  to  address  some  of  the  testing  delays.  In 
fiscal  year  2010.  the  Air  Force  increased  the  total 
number  of  aircraft  to  be  procured  from  54  to  77 
and  extended  planned  production  through  2018. 
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Production, 
design,  and 
technology 
maturity 


Design  and 
technology 
maturity 


Technology 

maturity 


Attainment  of  Product  Knowledge 

- * 


Development  COD  Dewlcpmeft  GAO 

start  design  sarttrrdjoxr  review 

(NA)  review  beason  (11/10) 

(NA)  (3/01) 
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Common  Name:  Global  Hawk 


Global  Hawk  Program 
Technology  Maturity 

The  critical  technologies  for  the  RQ-4B  are  mature. 
However,  the  program  must  still  successfully  test 
two  key  capabilities — the  advanced  signals 
intelligence  payload  and  multiple  platform  radar — to 
ensure  they  perform  as  expected.  The  first  flight  of 
an  RQ-4B  equipped  with  the  signals  intelligence 
payload  occurred  in  September  2008  and  operational 
testing  is  scheduled  to  be  completed  in  December 
2010.  After  delays  in  its  development,  the  first  flight 
of  an  RQ-1B  equipped  with  the  multiple  plat  form 
radar  is  expected  to  occur  in  April  201 1. 
Development  testing  is  underway. 

Design  Maturity 

The  RQ-4B  basic  airframe  design  is  stable  with  all  of 
its  expected  design  drawings  released;  however,  the 
program  remains  at  risk  for  late  design  changes  and 
costly  retrofits  if  problems  are  discovered  in  testing. 
During  the  first  year  of  production,  frequent 
substantive  engineering  changes  increased 
development  and  airframe  costs  and  delayed 
deliveries  and  testing.  Substantial  commonality 
between  the  RQ4A  and  RQ-4B  had  been  expected, 
but  as  the  designs  were  finalized  and  production 
geared  up,  the  design  differences  were  much  more 
extensive  and  complex  than  anticipated. 

Production  Maturity 

The  manufacturing  processes  for  the  RQ-4B 
airframe  are  mature  and  in  statistical  control.  In 
addition,  the  program  reports  that  it  is  meeting  its 
quality  goal  on  the  number  of  nonconforming  parts. 
The  RQ-4B  aircraft  is  being  produced  in  three 
configurations.  Block  20  aircraft  are  equipped  with 
an  enhanced  imagery'  intelligence  payload;  block  30 
aircraft  have  both  imagery  and  signals  intelligence 
payloads;  and  block  40  aircraft  will  have  an 
advanced  radar  surveillance  capability.  All  six  block 
20  aircraft  have  been  produced.  Production 
continues  on  block  30  and  block  40  aircraft.  The  first 
block  30  aircraft  was  delivered  in  November  2007 
and  the  first  block  40  aircraft  was  delivered  without 
the  sensor  in  November  2009. 

Other  Program  Issues 

The  Global  Hawk  program  expects  to  have  procured 
all  of  its  block  20  aircarft,  and  more  than  half  of  its 
block  30  and  block  40  aircraft  before  operational 
testing  is  complete.  As  a  result,  if  problems  are 
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found  in  operational  testing,  it  could  result  in  costly 
retrofits  for  large  numbers  of  aircraft.  The  Global 
Hawk  program  has  continued  to  experience  delays 
in  developmental  and  operational  testing.  The 
completion  of  operational  tests  for  the  block  20  and 
30  aircraft  has  been  delayed  nearly  4  years  to 
December  2010.  The  start  of  operational  testing  for 
block  40  aircraft  has  been  delayed  by  more  than  3 
years  to  March  2013.  According  to  the  Global  Hawk’s 
December  2009  Selected  Acquisition  Report,  several 
factors  contributed  to  the  most  recent  schedule 
slips,  including  developmental  test  discoveries; 
concurrent  development  and  production  testing; 
testing  resource  constraints;  and  weather  problems. 
According  to  program  officials,  a  shift  in  focus  and 
resources  required  to  address  a  Joint  Urgent 
Operational  Need,  using  two  block  20  aircraft,  has 
also  contributed  to  block  40  operational  test  delays. 
The  Air  Force  is  taking  steps  to  address  some  of  the 
testing  delays.  For  example,  the  program  is  now 
conducting  aircraft  acceptance  tests  at  Beale  Air 
Force  Base  in  order  to  free  up  resources  for 
operational  testing  at  Edwards  Air  Force  Base. 

Program  Office  Comments 

In  commenting  on  a  draft  of  this  assessment,  the  Air 
Force  emphasized  that  the  Global  Hawk  program 
has  improved  program  execution  while  reducing 
program  risk.  The  Air  Force  noted  that  older  RQ-4A 
Global  Haw'k  aircraft — which  we  did  not  assess — 
have  been  successfully  used  by  the  warfighters  and 
other  government  agencies  to  carry  out  various 
missions.  The  service  also  noted  that  each  of  the 
variants  of  its  larger  I1Q-1B  aircraft  is  now  either  in 
operations  or  testing  Flight  operations  of  deployed 
aircraft  and  flight  testing  of  the  advanced  radar 
payload  are  expected  to  begin  in  201 1.  The  Air  Force 
noted  that  current  challenges  facing  the  program 
include  initial  system  deployments  and 
normalization  of  operations  and  sustainment.  In 
addition  to  commenting  on  this  assessment,  the  Air 
Force  provided  technical  comments,  which  we 
incorporated  where  appropriate. 
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2003  -2012  Sponsored  Research  Topics 


Acquisition  Management 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  BCA:  Contractor  vs.  Organic  Growth 

■  Defense  Industry  Consolidation 

■  EU-US  Defense  Industrial  Relationships 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to 
Shipyard  Planning  Processes 

■  Managing  the  Services  Supply  Chain 

■  MOSA  Contracting  Implications 

■  Portfolio  Optimization  via  KVA  +  RO 

■  Private  Military  Sector 

■  Software  Requirements  for  OA 

■  Spiral  Development 

■  Strategy  for  Defense  Acquisition  Research 

■  The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 

Contract  Management 

■  Commodity  Sourcing  Strategies 

■  Contracting  Government  Procurement  Functions 

■  Contractors  in  21st-century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Model  for  Optimizing  Contingency  Contracting,  Planning  and  Execution 

■  Navy  Contract  Writing  Guide 

■  Past  Performance  in  Source  Selection 

■  Strategic  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  USAF  Energy  Savings  Performance  Contracts 

■  USAF  IT  Commodity  Council 

■  USMC  Contingency  Contracting 
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Financial  Management 


■  Acquisitions  via  Leasing:  MPS  case 

■  Budget  Scoring 

■  Budgeting  for  Capabilities-based  Planning 

■  Capital  Budgeting  for  the  DoD 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Financing  DoD  Budget  via  PPPs 

■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition 
Budgeting  Reform 

■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 

Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

■  Learning  Management  Systems 

■  Moral  Conduct  Waivers  and  First-term  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 

Logistics  Management 

■  Analysis  of  LAV  Depot  Maintenance 

■  Army  LOG  MOD 

■  ASDS  Product  Support  Analysis 

■  Cold-chain  Logistics 

■  Contractors  Supporting  Military  Operations 

■  Diffusion/Variability  on  Vendor  Performance  Evaluation 

■  Evolutionary  Acquisition 

■  Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 
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■  Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

■  Optimizing  CIWS  Lifecycle  Support  (LCS) 

■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance 
Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

-  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

-  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 


A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our 
website:  www.acquisitionresearch.net 
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